[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:05 dist-test-slave-kxfb krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:05 dist-test-slave-kxfb krb5kdc[395](info): set up 2 sockets
Sep 23 19:29:05 dist-test-slave-kxfb krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 19:29:06.519165 388 mini_kdc.cc:121] Time spent starting KDC: real 2.147s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260923 19:29:06.657928 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.138s user 0.001s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260923 19:29:06.989841 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.332s user 0.004s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260923 19:29:07.344684 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.355s user 0.002s sys 0.003s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:07 dist-test-slave-kxfb krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:07.454370 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46749
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:07.813879 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:07.814199 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:07.814321 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:07.821457 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:07.821569 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:07.821616 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:07.821697 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:07.821753 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:07.831602 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:07.833393 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:07.834810 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:07.847491 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:07.847584 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:07.847558 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:07.848675 408 server_base.cc:1034] running on GCE node
I20260923 19:29:07.849668 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:07.851624 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:07.852880 408 hybrid_clock.cc:648] HybridClock initialized: now 1790191747852783 us; error 86 us; skew 500 ppm
I20260923 19:29:07.853307 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:07 dist-test-slave-kxfb krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:07.857733 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:07.859611 408 webserver.cc:460] Webserver started at http://127.0.97.62:46341/ using document root <none> and password file <none>
I20260923 19:29:07.860471 408 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:07.860598 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:07.860939 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:07.864617 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "7f657a20b52e48a1a1663767ee4354a9"
format_stamp: "Formatted at 2026-09-23 19:29:07 on dist-test-slave-kxfb"
server_key: "a6f0f327824fb654e74ef98089f5dbdf"
server_key_iv: "bc28966d4f9996dd7dc4092085d9af45"
server_key_version: "encryptionkey@0"
I20260923 19:29:07.865468 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "7f657a20b52e48a1a1663767ee4354a9"
format_stamp: "Formatted at 2026-09-23 19:29:07 on dist-test-slave-kxfb"
server_key: "a6f0f327824fb654e74ef98089f5dbdf"
server_key_iv: "bc28966d4f9996dd7dc4092085d9af45"
server_key_version: "encryptionkey@0"
I20260923 19:29:07.871974 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 19:29:07.876281 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:07.878113 408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 19:29:07.878396 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "7f657a20b52e48a1a1663767ee4354a9"
format_stamp: "Formatted at 2026-09-23 19:29:07 on dist-test-slave-kxfb"
server_key: "a6f0f327824fb654e74ef98089f5dbdf"
server_key_iv: "bc28966d4f9996dd7dc4092085d9af45"
server_key_version: "encryptionkey@0"
I20260923 19:29:07.878616 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:07.897822 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:07.902833 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:07.903146 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:07.925504 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46749
I20260923 19:29:07.925555 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46749 every 8 connection(s)
I20260923 19:29:07.927129 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:07.931900 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:07.932885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 408
I20260923 19:29:07.933182 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:07.935727 388 external_mini_cluster.cc:1428] Setting key 8cdad90da8659c7ecd64d3aaa3dff1f5
I20260923 19:29:07.945353 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9: Bootstrap starting.
I20260923 19:29:07.948535 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:07.949536 475 log.cc:824] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:07.952350 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9: No bootstrap required, opened a new log
I20260923 19:29:07.962007 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f657a20b52e48a1a1663767ee4354a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260923 19:29:07.962407 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:07.962512 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f657a20b52e48a1a1663767ee4354a9, State: Initialized, Role: FOLLOWER
I20260923 19:29:07.963141 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f657a20b52e48a1a1663767ee4354a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260923 19:29:07.963346 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:07.963469 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:07.964147 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:07.966670 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f657a20b52e48a1a1663767ee4354a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260923 19:29:07.967166 475 leader_election.cc:302] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f657a20b52e48a1a1663767ee4354a9; no voters:
I20260923 19:29:07.967765 475 leader_election.cc:288] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:07.968042 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:07.969806 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [term 1 LEADER]: Becoming Leader. State: Replica: 7f657a20b52e48a1a1663767ee4354a9, State: Running, Role: LEADER
I20260923 19:29:07.969969 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 19:29:07 dist-test-slave-kxfb krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:07.970638 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f657a20b52e48a1a1663767ee4354a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260923 19:29:07.975919 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f657a20b52e48a1a1663767ee4354a9. Latest consensus state: current_term: 1 leader_uuid: "7f657a20b52e48a1a1663767ee4354a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f657a20b52e48a1a1663767ee4354a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } } }
I20260923 19:29:07.976460 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:07.977130 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f657a20b52e48a1a1663767ee4354a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f657a20b52e48a1a1663767ee4354a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } } }
I20260923 19:29:07.977427 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:07.987862 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:07.940899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.0.97.62:46749)
0923 19:29:07.942451 (+ 1552us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:07.942472 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:07.944564 (+ 2092us) server_negotiation.cc:407] Connection header received
0923 19:29:07.954728 (+ 10164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:07.954762 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:07.955326 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:07.955852 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:07.959339 (+ 3487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:07.961484 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:07.963584 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:07.964792 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:07.965014 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:07.972316 (+ 7302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:07.972328 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:07.972351 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:07.972421 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:07.978050 (+ 5629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:07.979404 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:07.979416 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:07.979426 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:07.979679 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:07.980306 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:07.980311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:07.980315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:07.981176 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:07.984841 (+ 3665us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:07.985157 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 19:29:07.985684 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":467,"threads_started":1}
I20260923 19:29:07.987964 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:08.000430 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:08.010215 487 catalog_manager.cc:1347] Generated new cluster ID: 0c5caac287d7443fa458991c15b14124
I20260923 19:29:08.010394 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:08.042115 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:08.044485 487 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:08.056672 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f657a20b52e48a1a1663767ee4354a9: Generated new TSK 0
I20260923 19:29:08.057698 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:29:08.118125 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:08.491178 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:08.491458 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:08.491550 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:08.498960 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:08.499073 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:08.499185 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:08.509112 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:08.510445 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:08.511741 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:08.525779 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:08.525874 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:08.526057 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:08.527038 501 server_base.cc:1034] running on GCE node
I20260923 19:29:08.527786 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:08.528885 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:08.530124 501 hybrid_clock.cc:648] HybridClock initialized: now 1790191748530058 us; error 53 us; skew 500 ppm
I20260923 19:29:08.530521 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:08 dist-test-slave-kxfb krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:08.533987 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:29:08.535758 501 webserver.cc:460] Webserver started at http://127.0.97.1:41449/ using document root <none> and password file <none>
I20260923 19:29:08.536558 501 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:08.536669 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:08.536963 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:08.540961 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "4f1978e80bc44eaea77da885c8b4c896"
format_stamp: "Formatted at 2026-09-23 19:29:08 on dist-test-slave-kxfb"
server_key: "8d963eadac6937b42968b1472522a50a"
server_key_iv: "58761e8da28c62268a0ef40c9b0db052"
server_key_version: "encryptionkey@0"
I20260923 19:29:08.541782 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4f1978e80bc44eaea77da885c8b4c896"
format_stamp: "Formatted at 2026-09-23 19:29:08 on dist-test-slave-kxfb"
server_key: "8d963eadac6937b42968b1472522a50a"
server_key_iv: "58761e8da28c62268a0ef40c9b0db052"
server_key_version: "encryptionkey@0"
I20260923 19:29:08.548002 501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 19:29:08.552131 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:08.553972 501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:29:08.554211 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "4f1978e80bc44eaea77da885c8b4c896"
format_stamp: "Formatted at 2026-09-23 19:29:08 on dist-test-slave-kxfb"
server_key: "8d963eadac6937b42968b1472522a50a"
server_key_iv: "58761e8da28c62268a0ef40c9b0db052"
server_key_version: "encryptionkey@0"
I20260923 19:29:08.554472 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:08.568357 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:08.573146 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:08.573449 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:08.574597 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:08.576773 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:08.576853 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:08.576974 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:08.577028 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:08.616534 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40231
I20260923 19:29:08.616586 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40231 every 8 connection(s)
I20260923 19:29:08.618007 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:08.628582 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 501
I20260923 19:29:08.628816 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:08.629803 388 external_mini_cluster.cc:1428] Setting key a7bc148786431d9e03429b6d0f088f20
Sep 23 19:29:08 dist-test-slave-kxfb krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:08.641091 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:08.621032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46795 (local address 127.0.97.62:46749)
0923 19:29:08.621554 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:08.621559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:08.622529 (+ 970us) server_negotiation.cc:407] Connection header received
0923 19:29:08.624594 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:08.624601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:08.624698 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:08.624801 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:08.626606 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:08.627397 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:08.630552 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:08.631046 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:08.631209 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:08.635634 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:08.635639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:08.635643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:08.635675 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:08.638072 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:08.638851 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:08.638855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:08.638857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:08.638973 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:08.639386 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:08.639390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:08.639392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:08.639649 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:08.639777 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:08.640519 (+ 742us) server_negotiation.cc:299] Negotiation successful
0923 19:29:08.640735 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":298,"threads_started":1}
I20260923 19:29:08.642134 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:08.621507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46749 (local address 127.0.97.1:46795)
0923 19:29:08.622337 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:08.622394 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:08.624285 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:08.625077 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:08.625087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:08.625714 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:08.626433 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:08.626454 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:08.627552 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:08.627556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:08.630344 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:08.630360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:08.631301 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:08.631307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:08.631530 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:08.632361 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:08.632393 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:08.635362 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:08.638224 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:08.638235 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:08.638252 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:08.638727 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:08.639076 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:08.639080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:08.639084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:08.639269 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:08.639768 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:08.639775 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:08.640159 (+ 384us) client_negotiation.cc:769] Sending connection context
0923 19:29:08.640393 (+ 234us) client_negotiation.cc:240] Negotiation successful
0923 19:29:08.640960 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":293,"threads_started":1}
I20260923 19:29:08.643683 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46749
I20260923 19:29:08.644174 629 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:08.645339 629 heartbeater.cc:511] Master 127.0.97.62:46749 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 19:29:08.647899 440 ts_manager.cc:194] Registered new tserver with Master: 4f1978e80bc44eaea77da885c8b4c896 (127.0.97.1:40231)
I20260923 19:29:08.649396 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:29:08.672065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:08.997330 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:08.997606 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:08.997701 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:09.004794 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:09.004904 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:09.005016 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:09.014885 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:09.016155 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:09.017386 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:09.030314 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:09.030409 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:09.030426 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:09.032152 635 server_base.cc:1034] running on GCE node
I20260923 19:29:09.032727 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:09.033670 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:09.034876 635 hybrid_clock.cc:648] HybridClock initialized: now 1790191749034824 us; error 51 us; skew 500 ppm
I20260923 19:29:09.035199 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:09 dist-test-slave-kxfb krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:09.038739 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:29:09.040235 635 webserver.cc:460] Webserver started at http://127.0.97.2:34305/ using document root <none> and password file <none>
I20260923 19:29:09.040912 635 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:09.041025 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:09.041313 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:09.044181 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "4fca6b9d238b429fb1b05a9d7761e7bd"
format_stamp: "Formatted at 2026-09-23 19:29:09 on dist-test-slave-kxfb"
server_key: "22f43cb14a40a8c6f18aea5d658e7be6"
server_key_iv: "7b4088c391d392ae946de584af1d0c38"
server_key_version: "encryptionkey@0"
I20260923 19:29:09.044917 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4fca6b9d238b429fb1b05a9d7761e7bd"
format_stamp: "Formatted at 2026-09-23 19:29:09 on dist-test-slave-kxfb"
server_key: "22f43cb14a40a8c6f18aea5d658e7be6"
server_key_iv: "7b4088c391d392ae946de584af1d0c38"
server_key_version: "encryptionkey@0"
I20260923 19:29:09.050268 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 19:29:09.053906 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:09.055455 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 19:29:09.055616 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "4fca6b9d238b429fb1b05a9d7761e7bd"
format_stamp: "Formatted at 2026-09-23 19:29:09 on dist-test-slave-kxfb"
server_key: "22f43cb14a40a8c6f18aea5d658e7be6"
server_key_iv: "7b4088c391d392ae946de584af1d0c38"
server_key_version: "encryptionkey@0"
I20260923 19:29:09.055796 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:09.077809 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:09.081898 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:09.082175 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:09.083254 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:09.084764 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:09.084841 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:09.084959 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:09.085009 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:09.122828 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41761
I20260923 19:29:09.123654 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41761 every 8 connection(s)
I20260923 19:29:09.124661 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:09.130009 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 635
I20260923 19:29:09.130168 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:09.130640 388 external_mini_cluster.cc:1428] Setting key 08de169b606a82ecdba0c0774fa451cc
Sep 23 19:29:09 dist-test-slave-kxfb krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:29:09.148243 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:09.127592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47973 (local address 127.0.97.62:46749)
0923 19:29:09.127842 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:09.127849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:09.128978 (+ 1129us) server_negotiation.cc:407] Connection header received
0923 19:29:09.131093 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:09.131099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:09.131198 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:09.131309 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:09.133539 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.134605 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.136305 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.136872 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.136994 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:09.141338 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:09.141344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:09.141348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:09.141384 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:09.144624 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.145679 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.145682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.145685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.145801 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.146353 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.146355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.146358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.146614 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:09.146718 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:09.147798 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0923 19:29:09.148003 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 19:29:09.149590 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:09.127977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46749 (local address 127.0.97.2:47973)
0923 19:29:09.128785 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:09.128835 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:09.130792 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:09.131654 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:09.131664 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:09.132486 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:09.133348 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:09.133370 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.134780 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.134784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:09.136147 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:09.136159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.137104 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.137109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:09.137355 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:09.138153 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:09.138182 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:09.141058 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:09.144814 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:09.144827 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:09.144854 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:09.145550 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:09.145910 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:09.145916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:09.145920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:09.146207 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:09.146746 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:09.146756 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:09.147315 (+ 559us) client_negotiation.cc:769] Sending connection context
0923 19:29:09.147675 (+ 360us) client_negotiation.cc:240] Negotiation successful
0923 19:29:09.148284 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":301,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 19:29:09.151438 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46749
I20260923 19:29:09.151870 763 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:09.152913 763 heartbeater.cc:511] Master 127.0.97.62:46749 requested a full tablet report, sending...
I20260923 19:29:09.154788 440 ts_manager.cc:194] Registered new tserver with Master: 4fca6b9d238b429fb1b05a9d7761e7bd (127.0.97.2:41761)
I20260923 19:29:09.155797 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:29:09.171943 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:09.488590 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:09.488868 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:09.488970 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:09.496130 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:09.496250 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:09.496367 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:09.506489 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46749
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:09.507809 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:09.509068 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:09.523660 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:09.524299 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:09.525930 768 server_base.cc:1034] running on GCE node
W20260923 19:29:09.525993 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:09.526919 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:09.528200 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:09.529460 768 hybrid_clock.cc:648] HybridClock initialized: now 1790191749529395 us; error 54 us; skew 500 ppm
I20260923 19:29:09.529896 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:09 dist-test-slave-kxfb krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:09.534160 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:29:09.535971 768 webserver.cc:460] Webserver started at http://127.0.97.3:33881/ using document root <none> and password file <none>
I20260923 19:29:09.536670 768 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:09.536773 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:09.537053 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:09.540055 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "46d81d0f76864d9fa31c2c106166849d"
format_stamp: "Formatted at 2026-09-23 19:29:09 on dist-test-slave-kxfb"
server_key: "a4aa315e709a932b255f5cf3cd5cc1dd"
server_key_iv: "958a7ea09531bf694047a090c2b36626"
server_key_version: "encryptionkey@0"
I20260923 19:29:09.540777 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "46d81d0f76864d9fa31c2c106166849d"
format_stamp: "Formatted at 2026-09-23 19:29:09 on dist-test-slave-kxfb"
server_key: "a4aa315e709a932b255f5cf3cd5cc1dd"
server_key_iv: "958a7ea09531bf694047a090c2b36626"
server_key_version: "encryptionkey@0"
I20260923 19:29:09.546397 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:29:09.550660 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:09.552295 768 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:29:09.552482 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "46d81d0f76864d9fa31c2c106166849d"
format_stamp: "Formatted at 2026-09-23 19:29:09 on dist-test-slave-kxfb"
server_key: "a4aa315e709a932b255f5cf3cd5cc1dd"
server_key_iv: "958a7ea09531bf694047a090c2b36626"
server_key_version: "encryptionkey@0"
I20260923 19:29:09.552672 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:09.580482 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:09.585494 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:09.585763 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:09.586781 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:09.588161 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:09.588239 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:09.588337 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:09.588393 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:09.629415 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39097
I20260923 19:29:09.629439 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39097 every 8 connection(s)
I20260923 19:29:09.630905 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:09.639873 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 768
I20260923 19:29:09.640091 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:09.640658 388 external_mini_cluster.cc:1428] Setting key 8e801b745ab0b9010f7576d9e776ebf7
Sep 23 19:29:09 dist-test-slave-kxfb krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:09.652369 629 heartbeater.cc:503] Master 127.0.97.62:46749 was elected leader, sending a full tablet report...
I20260923 19:29:09.655586 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:09.634461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36569 (local address 127.0.97.62:46749)
0923 19:29:09.634701 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:09.634708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:09.635977 (+ 1269us) server_negotiation.cc:407] Connection header received
0923 19:29:09.637964 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:09.637970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:09.638041 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:09.638145 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:09.640466 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.641254 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.643319 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.643924 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.644060 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:09.649024 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:09.649030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:09.649033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:09.649066 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:09.651746 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.652725 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.652732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.652736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.652966 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.653476 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.653481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.653484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.653918 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:09.654089 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:09.654968 (+ 879us) server_negotiation.cc:299] Negotiation successful
0923 19:29:09.655242 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 19:29:09.656536 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:09.634881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46749 (local address 127.0.97.3:36569)
0923 19:29:09.635774 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:09.635823 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:09.637672 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:09.638506 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:09.638522 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:09.639169 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:09.640218 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:09.640249 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.641438 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.641443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:09.643189 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:09.643208 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.644221 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.644231 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:09.644510 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:09.645369 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:09.645414 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:09.648764 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:09.651901 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:09.651912 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:09.651929 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:09.652537 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:09.653111 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:09.653116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:09.653119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:09.653339 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:09.654057 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:09.654063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:09.654568 (+ 505us) client_negotiation.cc:769] Sending connection context
0923 19:29:09.654820 (+ 252us) client_negotiation.cc:240] Negotiation successful
0923 19:29:09.655247 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":331,"threads_started":1}
I20260923 19:29:09.657886 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46749
I20260923 19:29:09.658177 896 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:09.659030 896 heartbeater.cc:511] Master 127.0.97.62:46749 requested a full tablet report, sending...
I20260923 19:29:09.660622 440 ts_manager.cc:194] Registered new tserver with Master: 46d81d0f76864d9fa31c2c106166849d (127.0.97.3:39097)
I20260923 19:29:09.661525 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36569
I20260923 19:29:09.669855 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 19:29:09 dist-test-slave-kxfb krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 19:29:09 dist-test-slave-kxfb krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:09.720152 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:09.702146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55016 (local address 127.0.97.1:40231)
0923 19:29:09.702935 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:09.702947 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:09.702993 (+ 46us) server_negotiation.cc:407] Connection header received
0923 19:29:09.703223 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:09.703234 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:09.703531 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:09.703697 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:09.706085 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.707234 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.708565 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.709189 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.709315 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:09.713691 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:09.713706 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:09.713725 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:09.713772 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:09.717100 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.717884 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.717893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.717896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.718033 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.718433 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.718438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.718440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.718839 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:09.719159 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:09.719459 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 19:29:09.719763 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":352,"threads_started":1}
W20260923 19:29:09.721688 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55016
Sep 23 19:29:09 dist-test-slave-kxfb krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 19:29:09 dist-test-slave-kxfb krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:09.769943 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:09.755624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.0.97.1:40231)
0923 19:29:09.755908 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:09.755914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:09.756083 (+ 169us) server_negotiation.cc:407] Connection header received
0923 19:29:09.756270 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:09.756275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:09.756345 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:09.756460 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:09.757980 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.758767 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.760204 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.760672 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.760776 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:09.764913 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:09.764919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:09.764924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:09.764967 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:09.767476 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.768179 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.768183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.768186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.768299 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.768624 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.768626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.768628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.768862 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:09.769035 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:09.769404 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 19:29:09.769603 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260923 19:29:09.771809 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 501
I20260923 19:29:09.785432 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:09.772835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.0.97.1:40231)
0923 19:29:09.773054 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:09.773060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:09.773342 (+ 282us) server_negotiation.cc:407] Connection header received
0923 19:29:09.773506 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:09.773511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:09.773579 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:09.773662 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:09.774908 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.775676 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.776968 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.777411 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.777505 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:09.780193 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:09.780199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:09.780202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:09.780239 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:09.782919 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.783598 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.783603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.783605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.783744 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.784105 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.784109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.784111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.784420 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:09.784608 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:09.784844 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 19:29:09.785017 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 19:29:09.786194 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:09.876313 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 501
I20260923 19:29:09.891934 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 635
Sep 23 19:29:09 dist-test-slave-kxfb krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:29:09.908857 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:09.893112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.0.97.2:41761)
0923 19:29:09.893658 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:09.893666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:09.893697 (+ 31us) server_negotiation.cc:407] Connection header received
0923 19:29:09.893764 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:09.893771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:09.894007 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:09.894146 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:09.895529 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.896578 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.897896 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:09.898640 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:09.898753 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:09.903137 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:09.903148 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:09.903163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:09.903198 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:09.905972 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.906838 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.906845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.906847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.906992 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:09.907431 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:09.907434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:09.907436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:09.907755 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:09.908016 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:09.908362 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 19:29:09.908567 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":268,"threads_started":1}
I20260923 19:29:09.910133 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:09.993438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 635
I20260923 19:29:10.014966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 768
Sep 23 19:29:10 dist-test-slave-kxfb krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:29:10.032246 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:10.016137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.0.97.3:39097)
0923 19:29:10.016869 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:10.016878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:10.016913 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:29:10.016985 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:10.016994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:10.017301 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:10.017476 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:10.018882 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:10.020002 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:10.021296 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:10.021903 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:10.022022 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:10.026597 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:10.026609 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:10.026627 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:10.026664 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:10.029545 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:10.030221 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:10.030228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:10.030230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:10.030404 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:10.030722 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:10.030725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:10.030727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:10.031123 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:10.031390 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:10.031657 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 19:29:10.031914 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":374,"threads_started":1}
I20260923 19:29:10.033675 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:10.117024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 768
I20260923 19:29:10.132606 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 408
I20260923 19:29:10.134016 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:10.236063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 408
2026-09-23T19:29:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5917 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:10 dist-test-slave-kxfb krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:10 dist-test-slave-kxfb krb5kdc[917](info): set up 2 sockets
Sep 23 19:29:10 dist-test-slave-kxfb krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260923 19:29:11.238251 388 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:11 dist-test-slave-kxfb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:11.446818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36127
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:11.754314 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:11.754590 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:11.754678 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:11.761597 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:11.761711 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:11.761757 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:11.761816 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:11.761864 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:11.771430 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:11.772680 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:11.773931 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:11.785997 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:11.785888 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:11.786172 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:11.787197 930 server_base.cc:1034] running on GCE node
I20260923 19:29:11.788024 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:11.789500 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:11.790711 930 hybrid_clock.cc:648] HybridClock initialized: now 1790191751790653 us; error 66 us; skew 500 ppm
I20260923 19:29:11.791064 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:11 dist-test-slave-kxfb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:11.798179 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:11.799846 930 webserver.cc:460] Webserver started at http://127.0.97.62:38097/ using document root <none> and password file <none>
I20260923 19:29:11.800542 930 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:11.800658 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:11.800935 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:11.803857 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "aac9e9c71d4a48f6bcfb284d17cb4318"
format_stamp: "Formatted at 2026-09-23 19:29:11 on dist-test-slave-kxfb"
server_key: "5ceb6638fae761347887f223b054af55"
server_key_iv: "3dc21970b3bec1ff0a16e47ce8471fb4"
server_key_version: "encryptionkey@0"
I20260923 19:29:11.804608 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "aac9e9c71d4a48f6bcfb284d17cb4318"
format_stamp: "Formatted at 2026-09-23 19:29:11 on dist-test-slave-kxfb"
server_key: "5ceb6638fae761347887f223b054af55"
server_key_iv: "3dc21970b3bec1ff0a16e47ce8471fb4"
server_key_version: "encryptionkey@0"
I20260923 19:29:11.810093 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 19:29:11.814080 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:11.815685 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:11.815881 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "aac9e9c71d4a48f6bcfb284d17cb4318"
format_stamp: "Formatted at 2026-09-23 19:29:11 on dist-test-slave-kxfb"
server_key: "5ceb6638fae761347887f223b054af55"
server_key_iv: "3dc21970b3bec1ff0a16e47ce8471fb4"
server_key_version: "encryptionkey@0"
I20260923 19:29:11.816079 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:11.832562 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:11.836813 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:11.837080 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:11.855403 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36127
I20260923 19:29:11.855472 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36127 every 8 connection(s)
I20260923 19:29:11.856719 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:11.861250 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:11.864830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 930
I20260923 19:29:11.864993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:11.865661 388 external_mini_cluster.cc:1428] Setting key 76c14c12d0cd4b1e52add8099a7e857f
I20260923 19:29:11.875044 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318: Bootstrap starting.
I20260923 19:29:11.879676 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318: Neither blocks nor log segments found. Creating new log.
Sep 23 19:29:11 dist-test-slave-kxfb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:11.881034 997 log.cc:824] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:11.884641 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318: No bootstrap required, opened a new log
I20260923 19:29:11.889636 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:11.867697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.0.97.62:36127)
0923 19:29:11.868502 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:11.868513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:11.868555 (+ 42us) server_negotiation.cc:407] Connection header received
0923 19:29:11.870424 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:11.870449 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:11.870876 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:11.871216 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:11.873181 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:11.874597 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:11.876365 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:11.877110 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:11.877249 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:11.882233 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:11.882242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:11.882260 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:11.882345 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:11.885428 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:11.886116 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:11.886122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:11.886129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:11.886337 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:11.886711 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:11.886716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:11.886723 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:11.887317 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:11.887608 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:11.888122 (+ 514us) server_negotiation.cc:299] Negotiation successful
0923 19:29:11.888541 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":371,"threads_started":1}
I20260923 19:29:11.894054 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9e9c71d4a48f6bcfb284d17cb4318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36127 } }
I20260923 19:29:11.894533 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:11.894639 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac9e9c71d4a48f6bcfb284d17cb4318, State: Initialized, Role: FOLLOWER
I20260923 19:29:11.895249 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9e9c71d4a48f6bcfb284d17cb4318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36127 } }
I20260923 19:29:11.895481 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:11.895630 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:11.895792 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:11.898378 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9e9c71d4a48f6bcfb284d17cb4318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36127 } }
I20260923 19:29:11.898970 997 leader_election.cc:302] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac9e9c71d4a48f6bcfb284d17cb4318; no voters:
I20260923 19:29:11.899618 997 leader_election.cc:288] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:11.899823 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:11.900619 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [term 1 LEADER]: Becoming Leader. State: Replica: aac9e9c71d4a48f6bcfb284d17cb4318, State: Running, Role: LEADER
I20260923 19:29:11.901262 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9e9c71d4a48f6bcfb284d17cb4318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36127 } }
I20260923 19:29:11.902068 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:11.904791 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac9e9c71d4a48f6bcfb284d17cb4318. Latest consensus state: current_term: 1 leader_uuid: "aac9e9c71d4a48f6bcfb284d17cb4318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9e9c71d4a48f6bcfb284d17cb4318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36127 } } }
I20260923 19:29:11.905354 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:11.904850 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac9e9c71d4a48f6bcfb284d17cb4318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9e9c71d4a48f6bcfb284d17cb4318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36127 } } }
I20260923 19:29:11.906407 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:11.906972 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:11.912354 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:11.920711 1009 catalog_manager.cc:1347] Generated new cluster ID: 9aeb532230824c7fb50b3c425bc8b2d6
I20260923 19:29:11.920828 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:11.937708 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:11.938999 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:11.944453 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac9e9c71d4a48f6bcfb284d17cb4318: Generated new TSK 0
I20260923 19:29:11.945199 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:29:12.002429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36127
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:12.322518 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:12.322816 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:12.322962 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:12.330392 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:12.330555 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:12.330732 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:12.341151 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36127
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:12.342552 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:12.344110 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:12.358049 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:12.358110 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:12.358074 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:12.359105 1023 server_base.cc:1034] running on GCE node
I20260923 19:29:12.359795 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:12.360790 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:12.362016 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790191752361945 us; error 69 us; skew 500 ppm
I20260923 19:29:12.362385 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:12 dist-test-slave-kxfb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:12.365722 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:29:12.367365 1023 webserver.cc:460] Webserver started at http://127.0.97.1:43785/ using document root <none> and password file <none>
I20260923 19:29:12.368109 1023 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:12.368225 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:12.368520 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:12.371418 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "f9f0574d559a453fa3a60bb0814fff80"
format_stamp: "Formatted at 2026-09-23 19:29:12 on dist-test-slave-kxfb"
server_key: "532b9cdda8ffbc77e55b0b7a198c76fb"
server_key_iv: "31026184095bef0379335fe4e2f927a2"
server_key_version: "encryptionkey@0"
I20260923 19:29:12.372184 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f9f0574d559a453fa3a60bb0814fff80"
format_stamp: "Formatted at 2026-09-23 19:29:12 on dist-test-slave-kxfb"
server_key: "532b9cdda8ffbc77e55b0b7a198c76fb"
server_key_iv: "31026184095bef0379335fe4e2f927a2"
server_key_version: "encryptionkey@0"
I20260923 19:29:12.377552 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:29:12.381230 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:12.382771 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:12.382929 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "f9f0574d559a453fa3a60bb0814fff80"
format_stamp: "Formatted at 2026-09-23 19:29:12 on dist-test-slave-kxfb"
server_key: "532b9cdda8ffbc77e55b0b7a198c76fb"
server_key_iv: "31026184095bef0379335fe4e2f927a2"
server_key_version: "encryptionkey@0"
I20260923 19:29:12.383141 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:12.399545 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:12.403564 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:12.403818 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:12.404780 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:12.406175 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:12.406273 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:12.406421 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:12.406476 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:12.440797 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35825
I20260923 19:29:12.440815 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35825 every 8 connection(s)
I20260923 19:29:12.442066 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:12.443229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 1023
I20260923 19:29:12.443504 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:12.444343 388 external_mini_cluster.cc:1428] Setting key 7901b6f782d5965dcf71215033a65cd1
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 23 19:29:12 dist-test-slave-kxfb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:12.472121 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:12.446644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48113 (local address 127.0.97.62:36127)
0923 19:29:12.447147 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:12.447166 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:12.451364 (+ 4198us) server_negotiation.cc:407] Connection header received
0923 19:29:12.453676 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:12.453683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:12.453790 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:12.453907 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:12.455920 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:12.456723 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:12.458455 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:12.458880 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:12.458997 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:12.465451 (+ 6454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:12.465458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:12.465463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:12.465514 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:12.468345 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:12.469333 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:12.469338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:12.469342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:12.469510 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:12.470029 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:12.470035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:12.470037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:12.470421 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:12.470573 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:12.471455 (+ 882us) server_negotiation.cc:299] Negotiation successful
0923 19:29:12.471726 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":268,"threads_started":1}
I20260923 19:29:12.473357 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:12.446125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36127 (local address 127.0.97.1:48113)
0923 19:29:12.451110 (+ 4985us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:12.451180 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:12.453271 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:12.454238 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:12.454249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:12.454981 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:12.455717 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:12.455737 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:12.456895 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:12.456899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:12.458262 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:12.458275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:12.459128 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:12.459135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:12.459405 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:12.460152 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:12.460178 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:12.465019 (+ 4841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:12.468576 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:12.468586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:12.468606 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:12.469155 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:12.469619 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:12.469623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:12.469625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:12.469863 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:12.470552 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:12.470561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:12.471005 (+ 444us) client_negotiation.cc:769] Sending connection context
0923 19:29:12.471277 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 19:29:12.471940 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4553,"thread_start_us":4285,"threads_started":1}
I20260923 19:29:12.475522 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36127
I20260923 19:29:12.475878 1151 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:12.476758 1151 heartbeater.cc:511] Master 127.0.97.62:36127 requested a full tablet report, sending...
I20260923 19:29:12.479251 962 ts_manager.cc:194] Registered new tserver with Master: f9f0574d559a453fa3a60bb0814fff80 (127.0.97.1:35825)
I20260923 19:29:12.480779 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:29:12.486198 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36127
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:12.798929 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:12.799224 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:12.799326 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:12.806890 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:12.807004 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:12.807154 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:12.817132 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36127
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:12.818498 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:12.819826 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:12.834475 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:12.834318 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:12.834378 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:12.835496 1157 server_base.cc:1034] running on GCE node
I20260923 19:29:12.836385 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:12.837698 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:12.838949 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790191752838898 us; error 46 us; skew 500 ppm
I20260923 19:29:12.839385 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:12 dist-test-slave-kxfb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:12.843210 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:29:12.844992 1157 webserver.cc:460] Webserver started at http://127.0.97.2:35087/ using document root <none> and password file <none>
I20260923 19:29:12.845837 1157 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:12.845965 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:12.846349 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:12.849891 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "dc85861ba14246a4b795907275384ad3"
format_stamp: "Formatted at 2026-09-23 19:29:12 on dist-test-slave-kxfb"
server_key: "696639ece6b866c4f72a1bdd45fe971b"
server_key_iv: "bb038011d85ad15b91f553068dd531d3"
server_key_version: "encryptionkey@0"
I20260923 19:29:12.850837 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dc85861ba14246a4b795907275384ad3"
format_stamp: "Formatted at 2026-09-23 19:29:12 on dist-test-slave-kxfb"
server_key: "696639ece6b866c4f72a1bdd45fe971b"
server_key_iv: "bb038011d85ad15b91f553068dd531d3"
server_key_version: "encryptionkey@0"
I20260923 19:29:12.858065 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260923 19:29:12.863987 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:12.866163 1157 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260923 19:29:12.866448 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "dc85861ba14246a4b795907275384ad3"
format_stamp: "Formatted at 2026-09-23 19:29:12 on dist-test-slave-kxfb"
server_key: "696639ece6b866c4f72a1bdd45fe971b"
server_key_iv: "bb038011d85ad15b91f553068dd531d3"
server_key_version: "encryptionkey@0"
I20260923 19:29:12.866724 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:12.894825 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:12.900393 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:12.900728 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:12.902064 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:12.903890 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:12.903998 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:12.904121 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:12.904203 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:12.943655 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35887
I20260923 19:29:12.943760 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35887 every 8 connection(s)
I20260923 19:29:12.945405 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:12.955565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 1157
I20260923 19:29:12.955878 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:12.956579 388 external_mini_cluster.cc:1428] Setting key 434c13c6cc924ceedd0031f76fd4bd31
Sep 23 19:29:12 dist-test-slave-kxfb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:12.970948 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:12.948403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53705 (local address 127.0.97.62:36127)
0923 19:29:12.948663 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:12.948671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:12.950063 (+ 1392us) server_negotiation.cc:407] Connection header received
0923 19:29:12.952560 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:12.952565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:12.952665 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:12.952792 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:12.955019 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:12.955892 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:12.958274 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:12.958956 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:12.959096 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:12.964169 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:12.964174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:12.964178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:12.964214 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:12.967254 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:12.968212 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:12.968217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:12.968221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:12.968368 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:12.968903 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:12.968907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:12.968910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:12.969228 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:12.969363 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:12.970329 (+ 966us) server_negotiation.cc:299] Negotiation successful
0923 19:29:12.970587 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 19:29:12.972374 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:12.948852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36127 (local address 127.0.97.2:53705)
0923 19:29:12.949806 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:12.949866 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:12.952134 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:12.953238 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:12.953249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:12.953927 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:12.954773 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:12.954796 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:12.956097 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:12.956104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:12.958085 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:12.958103 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:12.959309 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:12.959318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:12.959669 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:12.960575 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:12.960612 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:12.963832 (+ 3220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:12.967449 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:12.967459 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:12.967479 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:12.968039 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:12.968518 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:12.968522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:12.968525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:12.968756 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:12.969392 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:12.969400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:12.969863 (+ 463us) client_negotiation.cc:769] Sending connection context
0923 19:29:12.970163 (+ 300us) client_negotiation.cc:240] Negotiation successful
0923 19:29:12.970797 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":364,"threads_started":1}
I20260923 19:29:12.974170 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36127
I20260923 19:29:12.974653 1285 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:29:12.975694 1285 heartbeater.cc:511] Master 127.0.97.62:36127 requested a full tablet report, sending...
I20260923 19:29:12.977505 962 ts_manager.cc:194] Registered new tserver with Master: dc85861ba14246a4b795907275384ad3 (127.0.97.2:35887)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 19:29:12.978641 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:29:13.006733 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36127
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:13.318949 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:13.319345 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:13.319485 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:13.328799 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:13.328914 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:13.329079 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:13.338769 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36127
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:13.340124 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:13.341418 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:13.356092 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:13.356007 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:13.356042 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:13.356966 1290 server_base.cc:1034] running on GCE node
I20260923 19:29:13.357626 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:13.358651 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:13.359875 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790191753359821 us; error 48 us; skew 500 ppm
I20260923 19:29:13.360186 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:13 dist-test-slave-kxfb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:13.363322 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:29:13.364866 1290 webserver.cc:460] Webserver started at http://127.0.97.3:36227/ using document root <none> and password file <none>
I20260923 19:29:13.365531 1290 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:13.365655 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:13.365945 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:13.368842 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "c85fa0e6fe69481ca06b5894e70343fb"
format_stamp: "Formatted at 2026-09-23 19:29:13 on dist-test-slave-kxfb"
server_key: "1c93d92d5c40d5c780f4ac0ea9cbe776"
server_key_iv: "1d2e064a8df28ec6d86e1c445dfcf3e5"
server_key_version: "encryptionkey@0"
I20260923 19:29:13.369578 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c85fa0e6fe69481ca06b5894e70343fb"
format_stamp: "Formatted at 2026-09-23 19:29:13 on dist-test-slave-kxfb"
server_key: "1c93d92d5c40d5c780f4ac0ea9cbe776"
server_key_iv: "1d2e064a8df28ec6d86e1c445dfcf3e5"
server_key_version: "encryptionkey@0"
I20260923 19:29:13.374889 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 19:29:13.378511 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:13.380092 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:29:13.380259 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "c85fa0e6fe69481ca06b5894e70343fb"
format_stamp: "Formatted at 2026-09-23 19:29:13 on dist-test-slave-kxfb"
server_key: "1c93d92d5c40d5c780f4ac0ea9cbe776"
server_key_iv: "1d2e064a8df28ec6d86e1c445dfcf3e5"
server_key_version: "encryptionkey@0"
I20260923 19:29:13.380441 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:13.405346 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:13.409659 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:13.409929 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:13.411003 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:13.412405 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:13.412484 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:13.412591 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:13.412645 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:13.446144 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45115
I20260923 19:29:13.446269 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45115 every 8 connection(s)
I20260923 19:29:13.447431 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:13.448873 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 1290
I20260923 19:29:13.449028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:13.449476 388 external_mini_cluster.cc:1428] Setting key 36b9f307766affedaade862483e1cd5c
Sep 23 19:29:13 dist-test-slave-kxfb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:13.467957 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.450221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53561 (local address 127.0.97.62:36127)
0923 19:29:13.450432 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.450437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.451935 (+ 1498us) server_negotiation.cc:407] Connection header received
0923 19:29:13.453940 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.453945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.454025 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.454112 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:13.455630 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.456398 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.457897 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.458278 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.458416 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.462283 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:13.462315 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:13.462319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:13.462358 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:13.465021 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.465750 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.465755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.465758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.465875 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.466279 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.466300 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.466304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.466556 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:13.466699 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.467393 (+ 694us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.467600 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 19:29:13.468901 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.450844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36127 (local address 127.0.97.3:53561)
0923 19:29:13.451762 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:13.451820 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:13.453677 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:13.454406 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:13.454415 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:13.454912 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:13.455472 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.455488 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.456546 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.456552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.457759 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.457771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.458527 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.458534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.458751 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.459550 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:13.459578 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:13.462062 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:13.465170 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:13.465178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:13.465193 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:13.465630 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:13.465980 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:13.465984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:13.465986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:13.466170 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:13.466695 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:13.466704 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:13.467018 (+ 314us) client_negotiation.cc:769] Sending connection context
0923 19:29:13.467273 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 19:29:13.467788 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":350,"threads_started":1}
I20260923 19:29:13.470554 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36127
I20260923 19:29:13.470906 1418 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:13.471712 1418 heartbeater.cc:511] Master 127.0.97.62:36127 requested a full tablet report, sending...
I20260923 19:29:13.473317 962 ts_manager.cc:194] Registered new tserver with Master: c85fa0e6fe69481ca06b5894e70343fb (127.0.97.3:45115)
I20260923 19:29:13.474239 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53561
I20260923 19:29:13.477749 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:29:13.484360 1151 heartbeater.cc:503] Master 127.0.97.62:36127 was elected leader, sending a full tablet report...
I20260923 19:29:13.497974 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.484965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.0.97.62:36127)
0923 19:29:13.485234 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.485241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.485924 (+ 683us) server_negotiation.cc:407] Connection header received
0923 19:29:13.486161 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.486167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.486259 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.486403 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:13.488011 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.488756 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.490102 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.490562 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.490668 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.493128 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:13.493133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:13.493136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:13.493170 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:13.495634 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.496318 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.496322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.496324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.496427 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.496731 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.496734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.496736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.497013 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:13.497182 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.497533 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.497702 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 19:29:13.505577 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:29:13.514065 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:13.553359 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.528965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35887 (local address 127.0.0.1:33580)
0923 19:29:13.543015 (+ 14050us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:13.543047 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:13.543260 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:13.546011 (+ 2751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:13.546023 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:13.546083 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:13.546573 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.546587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.548500 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.548505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.550566 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.550580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.552390 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.552396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.552606 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.552659 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 19:29:13.552793 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 19:29:13.552993 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13852,"thread_start_us":10198,"threads_started":1}
I20260923 19:29:13.553498 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.529492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.0.97.2:35887)
0923 19:29:13.544926 (+ 15434us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.544938 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.544980 (+ 42us) server_negotiation.cc:407] Connection header received
0923 19:29:13.545117 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.545128 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.545451 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.546788 (+ 1337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:13.547035 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.548367 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.550780 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.552149 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.552308 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.552560 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.552893 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.553097 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15229,"thread_start_us":16158,"threads_started":1}
I20260923 19:29:13.555673 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.528556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35825 (local address 127.0.0.1:45090)
0923 19:29:13.539600 (+ 11044us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:13.539654 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:13.539869 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:13.546757 (+ 6888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:13.546762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:13.546786 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:13.547134 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.547145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.549576 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.549581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.552217 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.552232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.554706 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.554723 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.554900 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.554934 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:29:13.555022 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 19:29:13.555185 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10772,"mutex_wait_us":2,"thread_start_us":14820,"threads_started":1}
I20260923 19:29:13.556217 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.529626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.0.97.3:45115)
0923 19:29:13.545471 (+ 15845us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.545483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.546638 (+ 1155us) server_negotiation.cc:407] Connection header received
0923 19:29:13.546855 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.546863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.547070 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.547286 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:13.548300 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.549865 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.553084 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.554608 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.554728 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.554984 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.555169 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.555899 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15642,"thread_start_us":15455,"threads_started":1}
I20260923 19:29:13.556660 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.528867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.0.97.1:35825)
0923 19:29:13.546028 (+ 17161us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.546037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.546072 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:29:13.546194 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.546203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.546545 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.546783 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:13.547794 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.549416 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.552403 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.554184 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.554335 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.554566 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.556044 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.556231 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16995,"thread_start_us":16811,"threads_started":1}
I20260923 19:29:13.555740 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.528965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45115 (local address 127.0.0.1:42728)
0923 19:29:13.546452 (+ 17487us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:13.546472 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:13.546634 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:13.547602 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:13.547607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:13.547631 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:13.548072 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.548085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.550027 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.550033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.552878 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.552890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.554819 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.554826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.554991 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.555025 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:29:13.555110 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 19:29:13.555249 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17308,"mutex_wait_us":506,"thread_start_us":16714,"threads_started":1}
I20260923 19:29:13.558624 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 3af4c501e3ea4a1e956de6b251d76a7e (DEFAULT_TABLE table=test-table [id=34b4767cb44545e297d38ca8ba61b6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:13.560078 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af4c501e3ea4a1e956de6b251d76a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:13.561401 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 3af4c501e3ea4a1e956de6b251d76a7e (DEFAULT_TABLE table=test-table [id=34b4767cb44545e297d38ca8ba61b6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:13.561401 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 3af4c501e3ea4a1e956de6b251d76a7e (DEFAULT_TABLE table=test-table [id=34b4767cb44545e297d38ca8ba61b6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:13.563068 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af4c501e3ea4a1e956de6b251d76a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:13.563068 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af4c501e3ea4a1e956de6b251d76a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:13.574987 1436 tablet_bootstrap.cc:492] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3: Bootstrap starting.
I20260923 19:29:13.578346 1437 tablet_bootstrap.cc:492] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80: Bootstrap starting.
I20260923 19:29:13.578747 1436 tablet_bootstrap.cc:654] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:13.580274 1436 log.cc:824] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:13.582024 1438 tablet_bootstrap.cc:492] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb: Bootstrap starting.
I20260923 19:29:13.582547 1437 tablet_bootstrap.cc:654] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:13.584751 1436 tablet_bootstrap.cc:492] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3: No bootstrap required, opened a new log
I20260923 19:29:13.584830 1437 log.cc:824] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:13.585076 1436 ts_tablet_manager.cc:1397] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 19:29:13.588136 1437 tablet_bootstrap.cc:492] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80: No bootstrap required, opened a new log
I20260923 19:29:13.588454 1437 ts_tablet_manager.cc:1397] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260923 19:29:13.589301 1438 tablet_bootstrap.cc:654] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:13.590670 1438 log.cc:824] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:13.594772 1438 tablet_bootstrap.cc:492] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb: No bootstrap required, opened a new log
I20260923 19:29:13.595613 1438 ts_tablet_manager.cc:1397] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260923 19:29:13.596009 1436 raft_consensus.cc:348] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } }
I20260923 19:29:13.596503 1436 raft_consensus.cc:374] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:13.596603 1436 raft_consensus.cc:729] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc85861ba14246a4b795907275384ad3, State: Initialized, Role: FOLLOWER
I20260923 19:29:13.596815 1437 raft_consensus.cc:348] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } }
I20260923 19:29:13.597258 1437 raft_consensus.cc:374] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:13.597357 1437 raft_consensus.cc:729] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f0574d559a453fa3a60bb0814fff80, State: Initialized, Role: FOLLOWER
I20260923 19:29:13.597313 1436 consensus_queue.cc:260] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } }
I20260923 19:29:13.598100 1437 consensus_queue.cc:260] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } }
I20260923 19:29:13.599208 1285 heartbeater.cc:503] Master 127.0.97.62:36127 was elected leader, sending a full tablet report...
I20260923 19:29:13.599335 1436 ts_tablet_manager.cc:1428] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260923 19:29:13.601379 1437 ts_tablet_manager.cc:1428] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260923 19:29:13.606106 1438 raft_consensus.cc:348] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } }
I20260923 19:29:13.606535 1438 raft_consensus.cc:374] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:13.606635 1438 raft_consensus.cc:729] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85fa0e6fe69481ca06b5894e70343fb, State: Initialized, Role: FOLLOWER
I20260923 19:29:13.607237 1438 consensus_queue.cc:260] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } }
I20260923 19:29:13.608457 1418 heartbeater.cc:503] Master 127.0.97.62:36127 was elected leader, sending a full tablet report...
I20260923 19:29:13.609032 1438 ts_tablet_manager.cc:1428] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
W20260923 19:29:13.694540 1152 tablet.cc:2369] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:29:13.697846 1286 tablet.cc:2369] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:29:13.699065 1419 tablet.cc:2369] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:29:13.729950 1444 raft_consensus.cc:482] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:13.730266 1444 raft_consensus.cc:504] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } }
I20260923 19:29:13.732041 1444 leader_election.cc:288] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9f0574d559a453fa3a60bb0814fff80 (127.0.97.1:35825), dc85861ba14246a4b795907275384ad3 (127.0.97.2:35887)
I20260923 19:29:13.739713 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.732629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50281 (local address 127.0.97.2:35887)
0923 19:29:13.732972 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.732983 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.733601 (+ 618us) server_negotiation.cc:407] Connection header received
0923 19:29:13.733838 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.733845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.733938 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.734046 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:13.734542 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.735540 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.737542 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.738745 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.738855 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.738940 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.739378 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.739486 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260923 19:29:13.739789 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.732432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35887 (local address 127.0.97.3:50281)
0923 19:29:13.733483 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:13.733499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:13.733643 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:13.734109 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:13.734114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:13.734167 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:13.734390 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.734400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.735716 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.735720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.737370 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.737387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.738909 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.738915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.739176 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.739214 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 19:29:13.739328 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 19:29:13.739469 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":53,"spinlock_wait_cycles":25856}
I20260923 19:29:13.740852 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.732709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51383 (local address 127.0.97.1:35825)
0923 19:29:13.732917 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.732922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.734145 (+ 1223us) server_negotiation.cc:407] Connection header received
0923 19:29:13.734450 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.734463 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.734554 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.734675 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:13.735194 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.736081 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.738622 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.739808 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.739950 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.740056 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.740518 (+ 462us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.740656 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 19:29:13.740976 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.732467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35825 (local address 127.0.97.3:51383)
0923 19:29:13.734011 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:13.734033 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:13.734177 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:13.734728 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:13.734732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:13.734754 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:13.735034 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.735041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.736250 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.736257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.738405 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:13.738423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.740191 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.740195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:13.740342 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.740372 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 19:29:13.740470 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 19:29:13.740618 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"mutex_wait_us":1,"spinlock_wait_cycles":7936,"thread_start_us":1218,"threads_started":1}
I20260923 19:29:13.740895 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af4c501e3ea4a1e956de6b251d76a7e" candidate_uuid: "c85fa0e6fe69481ca06b5894e70343fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85861ba14246a4b795907275384ad3" is_pre_election: true
I20260923 19:29:13.741521 1240 raft_consensus.cc:2454] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c85fa0e6fe69481ca06b5894e70343fb in term 0.
I20260923 19:29:13.742017 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af4c501e3ea4a1e956de6b251d76a7e" candidate_uuid: "c85fa0e6fe69481ca06b5894e70343fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f0574d559a453fa3a60bb0814fff80" is_pre_election: true
I20260923 19:29:13.742319 1306 leader_election.cc:302] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c85fa0e6fe69481ca06b5894e70343fb, dc85861ba14246a4b795907275384ad3; no voters:
I20260923 19:29:13.742554 1106 raft_consensus.cc:2454] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c85fa0e6fe69481ca06b5894e70343fb in term 0.
I20260923 19:29:13.742803 1444 raft_consensus.cc:2781] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:13.743016 1444 raft_consensus.cc:482] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:13.743165 1444 raft_consensus.cc:3037] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:13.746618 1444 raft_consensus.cc:504] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } }
I20260923 19:29:13.747388 1444 leader_election.cc:288] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [CANDIDATE]: Term 1 election: Requested vote from peers f9f0574d559a453fa3a60bb0814fff80 (127.0.97.1:35825), dc85861ba14246a4b795907275384ad3 (127.0.97.2:35887)
I20260923 19:29:13.747879 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af4c501e3ea4a1e956de6b251d76a7e" candidate_uuid: "c85fa0e6fe69481ca06b5894e70343fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f0574d559a453fa3a60bb0814fff80"
I20260923 19:29:13.747972 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af4c501e3ea4a1e956de6b251d76a7e" candidate_uuid: "c85fa0e6fe69481ca06b5894e70343fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85861ba14246a4b795907275384ad3"
I20260923 19:29:13.748086 1106 raft_consensus.cc:3037] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:13.748229 1240 raft_consensus.cc:3037] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:13.750451 1106 raft_consensus.cc:2454] T 3af4c501e3ea4a1e956de6b251d76a7e P f9f0574d559a453fa3a60bb0814fff80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c85fa0e6fe69481ca06b5894e70343fb in term 1.
I20260923 19:29:13.750804 1240 raft_consensus.cc:2454] T 3af4c501e3ea4a1e956de6b251d76a7e P dc85861ba14246a4b795907275384ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c85fa0e6fe69481ca06b5894e70343fb in term 1.
I20260923 19:29:13.750921 1309 leader_election.cc:302] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c85fa0e6fe69481ca06b5894e70343fb, f9f0574d559a453fa3a60bb0814fff80; no voters:
I20260923 19:29:13.751328 1444 raft_consensus.cc:2781] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:13.751835 1444 raft_consensus.cc:686] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [term 1 LEADER]: Becoming Leader. State: Replica: c85fa0e6fe69481ca06b5894e70343fb, State: Running, Role: LEADER
I20260923 19:29:13.752480 1444 consensus_queue.cc:237] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } }
I20260923 19:29:13.757889 961 catalog_manager.cc:5495] T 3af4c501e3ea4a1e956de6b251d76a7e P c85fa0e6fe69481ca06b5894e70343fb reported cstate change: term changed from 0 to 1, leader changed from <none> to c85fa0e6fe69481ca06b5894e70343fb (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c85fa0e6fe69481ca06b5894e70343fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f0574d559a453fa3a60bb0814fff80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c85fa0e6fe69481ca06b5894e70343fb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc85861ba14246a4b795907275384ad3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35887 } health_report { overall_health: UNKNOWN } } }
Sep 23 19:29:13 dist-test-slave-kxfb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:13.802667 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.786144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.0.97.1:35825)
0923 19:29:13.786350 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.786355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.787703 (+ 1348us) server_negotiation.cc:407] Connection header received
0923 19:29:13.787865 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.787872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.787979 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.788083 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:13.789557 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.790397 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.791695 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.792237 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.792358 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.796390 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:13.796406 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:13.796427 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:13.796474 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:13.799579 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.800349 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.800357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.800360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.800535 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.800953 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.800957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.800959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.801386 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:13.801660 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.801947 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.802151 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 19:29:13 dist-test-slave-kxfb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 19:29:13 dist-test-slave-kxfb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:13.848291 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.834078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.0.97.1:35825)
0923 19:29:13.834444 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.834451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.834708 (+ 257us) server_negotiation.cc:407] Connection header received
0923 19:29:13.834889 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.834894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.834977 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.835078 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:13.836378 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.837231 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.838613 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.839093 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.839198 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.843164 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:13.843172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:13.843176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:13.843234 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:13.845828 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.846628 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.846633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.846635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.846753 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.847096 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.847099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.847101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.847382 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:13.847591 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.847835 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.848012 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
Sep 23 19:29:13 dist-test-slave-kxfb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 19:29:13 dist-test-slave-kxfb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:13.897053 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.883451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.0.97.1:35825)
0923 19:29:13.883695 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.883701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.883936 (+ 235us) server_negotiation.cc:407] Connection header received
0923 19:29:13.884149 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.884154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.884219 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.884308 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:13.885599 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.886360 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.887593 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.888000 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.888106 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.892162 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:13.892168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:13.892171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:13.892212 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:13.894739 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.895431 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.895435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.895437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.895560 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.895869 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.895871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.895873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.896154 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:13.896326 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.896616 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.896782 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260923 19:29:13.899375 1066 tablet_service.cc:2908] Rejecting scan request for tablet 3af4c501e3ea4a1e956de6b251d76a7e: Uninitialized: clean time has not yet been initialized
I20260923 19:29:13.902756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1023
I20260923 19:29:13.916828 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:13.903775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.0.97.1:35825)
0923 19:29:13.903982 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:13.903989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:13.904349 (+ 360us) server_negotiation.cc:407] Connection header received
0923 19:29:13.904496 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:13.904500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:13.904571 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:13.904647 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:13.905900 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.906670 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.907941 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:13.908454 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:13.908617 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:13.911762 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:13.911767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:13.911771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:13.911808 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:13.914269 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.914927 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.914933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.914937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.915061 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:13.915409 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:13.915414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:13.915418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:13.915714 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:13.915995 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:13.916276 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 19:29:13.916497 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 19:29:13.917526 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:14.016319 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 1023
I20260923 19:29:14.032850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1157
Sep 23 19:29:14 dist-test-slave-kxfb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:29:14.048801 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:14.033835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.0.97.2:35887)
0923 19:29:14.034027 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:14.034032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:14.034055 (+ 23us) server_negotiation.cc:407] Connection header received
0923 19:29:14.034124 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:14.034128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:14.034226 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:14.034386 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:14.035553 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:14.036400 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:14.037748 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:14.038159 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:14.038253 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:14.042281 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:14.042330 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:14.042357 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:14.042395 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:14.045356 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:14.046099 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:14.046108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:14.046111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:14.046266 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:14.046944 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:14.046951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:14.046956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:14.047581 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:14.047893 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:14.048254 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 19:29:14.048469 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 19:29:14.049649 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:14.137522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 1157
I20260923 19:29:14.152823 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1290
Sep 23 19:29:14 dist-test-slave-kxfb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:29:14.168370 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:14.153733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.0.97.3:45115)
0923 19:29:14.154258 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:14.154263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:14.154283 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:29:14.154413 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:14.154418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:14.154500 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:14.154605 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:14.155831 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:14.156585 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:14.157839 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:14.158354 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:14.158468 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:14.162419 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:14.162435 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:14.162457 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:14.162496 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:14.165444 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:14.166069 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:14.166077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:14.166080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:14.166263 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:14.166847 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:14.166850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:14.166853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:14.167221 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:14.167500 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:14.167712 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 19:29:14.167901 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":284,"threads_started":1}
I20260923 19:29:14.169193 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:14.264807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 1290
I20260923 19:29:14.278997 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 930
I20260923 19:29:14.280478 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:14.399842 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 930
2026-09-23T19:29:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4162 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:14 dist-test-slave-kxfb krb5kdc[1469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:14 dist-test-slave-kxfb krb5kdc[1469](info): set up 2 sockets
Sep 23 19:29:14 dist-test-slave-kxfb krb5kdc[1469](info): commencing operation
krb5kdc: starting...
W20260923 19:29:15.406617 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:15 dist-test-slave-kxfb krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:15.607879 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38587
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:15.922457 1482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:15.922767 1482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:15.922879 1482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:15.930379 1482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:15.930513 1482 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 19:29:15.930581 1482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:15.930665 1482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:15.930725 1482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:15.930810 1482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:15.940706 1482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38587
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:15.941985 1482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:15.943327 1482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:15.955525 1488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:15.955456 1490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:15.956025 1487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:15.956926 1482 server_base.cc:1034] running on GCE node
I20260923 19:29:15.957934 1482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:15.959779 1482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:15.961040 1482 hybrid_clock.cc:648] HybridClock initialized: now 1790191755960966 us; error 76 us; skew 500 ppm
I20260923 19:29:15.961453 1482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:15 dist-test-slave-kxfb krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:15.965178 1482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:15.967024 1482 webserver.cc:460] Webserver started at http://127.0.97.62:33717/ using document root <none> and password file <none>
I20260923 19:29:15.967882 1482 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:15.967986 1482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:15.968353 1482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:15.971783 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "8eab7a2276ee4d358242e075c50cc275"
format_stamp: "Formatted at 2026-09-23 19:29:15 on dist-test-slave-kxfb"
server_key: "2324a2d2a082bb136e21961235d89f26"
server_key_iv: "778165981c5d7a6e7dfdb756ff1811f0"
server_key_version: "encryptionkey@0"
I20260923 19:29:15.972684 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "8eab7a2276ee4d358242e075c50cc275"
format_stamp: "Formatted at 2026-09-23 19:29:15 on dist-test-slave-kxfb"
server_key: "2324a2d2a082bb136e21961235d89f26"
server_key_iv: "778165981c5d7a6e7dfdb756ff1811f0"
server_key_version: "encryptionkey@0"
I20260923 19:29:15.978742 1482 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 19:29:15.983150 1497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:15.984905 1482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:29:15.985114 1482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "8eab7a2276ee4d358242e075c50cc275"
format_stamp: "Formatted at 2026-09-23 19:29:15 on dist-test-slave-kxfb"
server_key: "2324a2d2a082bb136e21961235d89f26"
server_key_iv: "778165981c5d7a6e7dfdb756ff1811f0"
server_key_version: "encryptionkey@0"
I20260923 19:29:15.985380 1482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:16.003893 1482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:16.008973 1482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:16.009246 1482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:16.033550 1482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38587
I20260923 19:29:16.033520 1558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38587 every 8 connection(s)
I20260923 19:29:16.035882 1482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:16.036957 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 1482
I20260923 19:29:16.037170 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:16.037981 388 external_mini_cluster.cc:1428] Setting key 090e88f88aa89139440bbc381ff2b50c
I20260923 19:29:16.041700 1559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 19:29:16 dist-test-slave-kxfb krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:16.054819 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275: Bootstrap starting.
I20260923 19:29:16.059015 1559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:16.060532 1559 log.cc:824] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:16.063706 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:16.042057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.0.97.62:38587)
0923 19:29:16.042899 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:16.042912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:16.042962 (+ 50us) server_negotiation.cc:407] Connection header received
0923 19:29:16.044679 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:16.044702 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:16.045149 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:16.045540 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:16.046985 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.048417 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.049642 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.050417 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.050609 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:16.055028 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:16.055038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:16.055058 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:16.055102 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:16.058547 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:16.059349 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:16.059363 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:16.059377 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:16.059663 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:16.060111 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:16.060116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:16.060120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:16.060932 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:16.061311 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:16.061676 (+ 365us) server_negotiation.cc:299] Negotiation successful
0923 19:29:16.062174 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":394,"threads_started":1}
I20260923 19:29:16.064571 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275: No bootstrap required, opened a new log
I20260923 19:29:16.073493 1559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab7a2276ee4d358242e075c50cc275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38587 } }
I20260923 19:29:16.073939 1559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:16.074044 1559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eab7a2276ee4d358242e075c50cc275, State: Initialized, Role: FOLLOWER
I20260923 19:29:16.074775 1559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab7a2276ee4d358242e075c50cc275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38587 } }
I20260923 19:29:16.075011 1559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:16.075153 1559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:16.075342 1559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:16.077826 1559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab7a2276ee4d358242e075c50cc275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38587 } }
I20260923 19:29:16.078501 1559 leader_election.cc:302] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eab7a2276ee4d358242e075c50cc275; no voters:
I20260923 19:29:16.079177 1559 leader_election.cc:288] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:16.079380 1564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:16.080376 1564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [term 1 LEADER]: Becoming Leader. State: Replica: 8eab7a2276ee4d358242e075c50cc275, State: Running, Role: LEADER
I20260923 19:29:16.081033 1564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab7a2276ee4d358242e075c50cc275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38587 } }
I20260923 19:29:16.081713 1559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:16.084427 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eab7a2276ee4d358242e075c50cc275. Latest consensus state: current_term: 1 leader_uuid: "8eab7a2276ee4d358242e075c50cc275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab7a2276ee4d358242e075c50cc275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38587 } } }
I20260923 19:29:16.084511 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eab7a2276ee4d358242e075c50cc275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab7a2276ee4d358242e075c50cc275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38587 } } }
I20260923 19:29:16.084841 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:16.084908 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:16.087231 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:16.093731 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:16.103869 1570 catalog_manager.cc:1347] Generated new cluster ID: 6c0ca2f6307c4441a0630cd364eef889
I20260923 19:29:16.104013 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:16.141517 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:16.143550 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:16.153805 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275: Generated new TSK 0
I20260923 19:29:16.154920 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:29:16.203040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38587
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:16.569512 1585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:16.569870 1585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:16.570001 1585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:16.577898 1585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:16.578055 1585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:16.578174 1585 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 19:29:16.578492 1585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:16.590226 1585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:16.591965 1585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:16.593605 1585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:16.609448 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:16.609541 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:16.609630 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:16.616052 1585 server_base.cc:1034] running on GCE node
I20260923 19:29:16.616895 1585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:16.618028 1585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:16.619279 1585 hybrid_clock.cc:648] HybridClock initialized: now 1790191756619177 us; error 82 us; skew 500 ppm
I20260923 19:29:16.619630 1585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:16 dist-test-slave-kxfb krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:16.623147 1585 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:29:16.624920 1585 webserver.cc:460] Webserver started at http://127.0.97.1:41741/ using document root <none> and password file <none>
I20260923 19:29:16.625720 1585 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:16.625818 1585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:16.626132 1585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:16.629457 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "3f9bad2d029f4eecad0445cfba260bb5"
format_stamp: "Formatted at 2026-09-23 19:29:16 on dist-test-slave-kxfb"
server_key: "d583d1038f16a181ceb9b13bf017c13a"
server_key_iv: "d4d792932bf83c90fd2855f8574f8486"
server_key_version: "encryptionkey@0"
I20260923 19:29:16.630347 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3f9bad2d029f4eecad0445cfba260bb5"
format_stamp: "Formatted at 2026-09-23 19:29:16 on dist-test-slave-kxfb"
server_key: "d583d1038f16a181ceb9b13bf017c13a"
server_key_iv: "d4d792932bf83c90fd2855f8574f8486"
server_key_version: "encryptionkey@0"
I20260923 19:29:16.636514 1585 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 19:29:16.640807 1600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:16.642660 1585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:29:16.642854 1585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "3f9bad2d029f4eecad0445cfba260bb5"
format_stamp: "Formatted at 2026-09-23 19:29:16 on dist-test-slave-kxfb"
server_key: "d583d1038f16a181ceb9b13bf017c13a"
server_key_iv: "d4d792932bf83c90fd2855f8574f8486"
server_key_version: "encryptionkey@0"
I20260923 19:29:16.643096 1585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:16.674147 1585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:16.679013 1585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:16.679307 1585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:16.682413 1585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:16.682503 1585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:16.682628 1585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:16.682693 1585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 19:29:16 dist-test-slave-kxfb krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:16.707715 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:16.683213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34271 (local address 127.0.97.62:38587)
0923 19:29:16.683910 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:16.683918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:16.685394 (+ 1476us) server_negotiation.cc:407] Connection header received
0923 19:29:16.687716 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:16.687722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:16.687811 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:16.687923 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:16.690606 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.691917 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.693798 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.694504 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.694642 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:16.700714 (+ 6072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:16.700722 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:16.700727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:16.700779 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:16.704212 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:16.705203 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:16.705209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:16.705213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:16.705342 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:16.705776 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:16.705779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:16.705781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:16.706068 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:16.706210 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:16.707128 (+ 918us) server_negotiation.cc:299] Negotiation successful
0923 19:29:16.707359 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":409,"threads_started":1}
I20260923 19:29:16.708923 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:16.684246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38587 (local address 127.0.97.1:34271)
0923 19:29:16.685188 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:16.685247 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:16.687395 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:16.688174 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:16.688191 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:16.689087 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:16.690259 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:16.690309 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.692221 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.692225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:16.693616 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:16.693628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.694741 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.694746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:16.694953 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:16.695832 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:16.695873 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:16.700278 (+ 4405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:16.704428 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:16.704438 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:16.704460 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:16.705021 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:16.705460 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:16.705465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:16.705469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:16.705657 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:16.706209 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:16.706217 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:16.706737 (+ 520us) client_negotiation.cc:769] Sending connection context
0923 19:29:16.707014 (+ 277us) client_negotiation.cc:240] Negotiation successful
0923 19:29:16.707574 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":383,"threads_started":1}
I20260923 19:29:16.729331 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:16.714906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38587 (local address 127.0.97.1:41081)
0923 19:29:16.715571 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:16.715588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:16.715735 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:16.716028 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:16.716033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:16.716645 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:16.717091 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:16.717107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.718361 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.718367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:16.719846 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:16.719864 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.720978 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.720985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:16.721295 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:16.721945 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:16.721960 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:16.722904 (+ 944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:16.726337 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:16.726343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:16.726350 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:16.726888 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:16.727298 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:16.727302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:16.727310 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:16.727476 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:16.728422 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:16.728427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:16.728631 (+ 204us) client_negotiation.cc:769] Sending connection context
0923 19:29:16.728747 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 19:29:16.728988 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":371,"threads_started":1}
I20260923 19:29:16.729710 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:16.714772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41081 (local address 127.0.97.62:38587)
0923 19:29:16.714916 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:16.714919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:16.715683 (+ 764us) server_negotiation.cc:407] Connection header received
0923 19:29:16.715839 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:16.715844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:16.715901 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:16.715975 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:16.717279 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.718206 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.720046 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:16.720581 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:16.720712 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:16.723132 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:16.723138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:16.723142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:16.723177 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:16.726137 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:16.727051 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:16.727058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:16.727061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:16.727187 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:16.727692 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:16.727702 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:16.727706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:16.728245 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:16.728454 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:16.728928 (+ 474us) server_negotiation.cc:299] Negotiation successful
0923 19:29:16.729262 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 19:29:16.742730 1585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33059
I20260923 19:29:16.742787 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33059 every 8 connection(s)
I20260923 19:29:16.744160 1585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:16.745512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 1585
I20260923 19:29:16.745699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:16.746374 388 external_mini_cluster.cc:1428] Setting key ffa9fb29a53c8babe4939b11da3deb10
I20260923 19:29:16.747601 1721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38587
I20260923 19:29:16.747857 1721 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:16.748679 1721 heartbeater.cc:511] Master 127.0.97.62:38587 requested a full tablet report, sending...
I20260923 19:29:16.750828 1514 ts_manager.cc:194] Registered new tserver with Master: 3f9bad2d029f4eecad0445cfba260bb5 (127.0.97.1:33059)
I20260923 19:29:16.752417 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34271
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:29:16.786048 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38587
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:17.117674 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:17.117990 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:17.118110 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:17.125366 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:17.125499 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:17.125615 1725 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 19:29:17.125725 1725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:17.135653 1725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:17.136961 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:17.138265 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:17.152365 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:17.152441 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:17.152393 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:17.153649 1725 server_base.cc:1034] running on GCE node
I20260923 19:29:17.154413 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:17.155434 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:17.156672 1725 hybrid_clock.cc:648] HybridClock initialized: now 1790191757156585 us; error 60 us; skew 500 ppm
I20260923 19:29:17.157018 1725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:17 dist-test-slave-kxfb krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:17.162961 1725 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:29:17.164593 1725 webserver.cc:460] Webserver started at http://127.0.97.2:39697/ using document root <none> and password file <none>
I20260923 19:29:17.165343 1725 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:17.165429 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:17.165688 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:17.168571 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "cd91bad592234245bcfe50254d56b4ce"
format_stamp: "Formatted at 2026-09-23 19:29:17 on dist-test-slave-kxfb"
server_key: "8b4005c11a8d5b0bcdb24580e40ea38b"
server_key_iv: "d036ee838a118462cdab64af3ea94bdd"
server_key_version: "encryptionkey@0"
I20260923 19:29:17.169303 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cd91bad592234245bcfe50254d56b4ce"
format_stamp: "Formatted at 2026-09-23 19:29:17 on dist-test-slave-kxfb"
server_key: "8b4005c11a8d5b0bcdb24580e40ea38b"
server_key_iv: "d036ee838a118462cdab64af3ea94bdd"
server_key_version: "encryptionkey@0"
I20260923 19:29:17.174665 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 19:29:17.178351 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:17.179912 1725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:17.180126 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "cd91bad592234245bcfe50254d56b4ce"
format_stamp: "Formatted at 2026-09-23 19:29:17 on dist-test-slave-kxfb"
server_key: "8b4005c11a8d5b0bcdb24580e40ea38b"
server_key_iv: "d036ee838a118462cdab64af3ea94bdd"
server_key_version: "encryptionkey@0"
I20260923 19:29:17.180336 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:17.197193 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:17.201462 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:17.201727 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:17.204867 1725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:17.204952 1725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:17.205113 1725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:17.205190 1725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 19:29:17 dist-test-slave-kxfb krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:17.233757 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.206585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38587 (local address 127.0.97.2:50459)
0923 19:29:17.207609 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:17.207659 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:17.210269 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:17.211363 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:17.211377 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:17.212499 (+ 1122us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:17.213677 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.213714 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.215284 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.215290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.217899 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.217923 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.219105 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.219113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.219473 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.220337 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:17.220370 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:17.223990 (+ 3620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:17.228122 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:17.228135 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:17.228166 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:17.228944 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:17.229384 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:17.229389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:17.229393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:17.229658 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:17.230282 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:17.230322 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:17.230908 (+ 586us) client_negotiation.cc:769] Sending connection context
0923 19:29:17.231318 (+ 410us) client_negotiation.cc:240] Negotiation successful
0923 19:29:17.232020 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":414,"threads_started":1}
I20260923 19:29:17.234899 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.205947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50459 (local address 127.0.97.62:38587)
0923 19:29:17.206137 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:17.206142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:17.207819 (+ 1677us) server_negotiation.cc:407] Connection header received
0923 19:29:17.210709 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:17.210715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:17.210819 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:17.210937 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:17.213956 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.215052 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.218108 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.218789 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.218975 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.224373 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:17.224382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:17.224388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:17.224439 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:17.227934 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.229132 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.229137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.229139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.229258 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.229804 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.229808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.229811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.230143 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:17.230271 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:17.234246 (+ 3975us) server_negotiation.cc:299] Negotiation successful
0923 19:29:17.234561 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 19:29:17.257570 1725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36075
I20260923 19:29:17.257671 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36075 every 8 connection(s)
I20260923 19:29:17.259114 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:17.261651 1860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38587
I20260923 19:29:17.261976 1860 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:17.262140 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.244203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38587 (local address 127.0.97.2:36827)
0923 19:29:17.245360 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:17.245379 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:17.245570 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:17.246133 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:17.246138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:17.246898 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:17.247285 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.247297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.248489 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.248494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.250316 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.250331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.251570 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.251577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.251771 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.252750 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:17.252775 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:17.254333 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:17.258226 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:17.258233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:17.258238 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:17.258887 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:17.259405 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:17.259409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:17.259412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:17.259586 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:17.260451 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:17.260455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:17.260692 (+ 237us) client_negotiation.cc:769] Sending connection context
0923 19:29:17.260863 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 19:29:17.261587 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":643,"threads_started":1}
I20260923 19:29:17.262420 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.244163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36827 (local address 127.0.97.62:38587)
0923 19:29:17.244538 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:17.244545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:17.245497 (+ 952us) server_negotiation.cc:407] Connection header received
0923 19:29:17.245706 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:17.245712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:17.245815 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:17.245926 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:17.247440 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.248345 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.250522 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.251314 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.251487 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.254630 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:17.254638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:17.254643 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:17.254689 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:17.258051 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.259041 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.259045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.259048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.259171 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.259747 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.259752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.259754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.260229 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:17.260388 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:17.261805 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0923 19:29:17.262055 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260923 19:29:17.262979 1860 heartbeater.cc:511] Master 127.0.97.62:38587 requested a full tablet report, sending...
I20260923 19:29:17.264554 1514 ts_manager.cc:194] Registered new tserver with Master: cd91bad592234245bcfe50254d56b4ce (127.0.97.2:36075)
I20260923 19:29:17.265065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 1725
I20260923 19:29:17.265245 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:17.265551 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50459
I20260923 19:29:17.265758 388 external_mini_cluster.cc:1428] Setting key a16a2feb30a77121e7986faace2489a1
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:29:17.306129 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38587
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:17.671698 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:17.672053 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:17.672163 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:17.679849 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:17.679992 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:17.680099 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 19:29:17.680227 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:17.691166 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:17.692799 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:17.694378 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:17.709551 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:17.709558 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:17.709592 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:17.710840 1864 server_base.cc:1034] running on GCE node
I20260923 19:29:17.711582 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:17.712719 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:17.713989 1864 hybrid_clock.cc:648] HybridClock initialized: now 1790191757713892 us; error 75 us; skew 500 ppm
I20260923 19:29:17.714411 1864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:17 dist-test-slave-kxfb krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:17.717981 1864 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:29:17.719791 1864 webserver.cc:460] Webserver started at http://127.0.97.3:42177/ using document root <none> and password file <none>
I20260923 19:29:17.720624 1864 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:17.720731 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:17.721096 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:17.724596 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "553185031bf44b6499eb1441a350dd91"
format_stamp: "Formatted at 2026-09-23 19:29:17 on dist-test-slave-kxfb"
server_key: "0587f84c183498d2b2e4a787e33087cb"
server_key_iv: "e0288f4b5570a0db15ce3d9286d4a48b"
server_key_version: "encryptionkey@0"
I20260923 19:29:17.725485 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "553185031bf44b6499eb1441a350dd91"
format_stamp: "Formatted at 2026-09-23 19:29:17 on dist-test-slave-kxfb"
server_key: "0587f84c183498d2b2e4a787e33087cb"
server_key_iv: "e0288f4b5570a0db15ce3d9286d4a48b"
server_key_version: "encryptionkey@0"
I20260923 19:29:17.731612 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 19:29:17.735908 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:17.737687 1864 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:29:17.737864 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "553185031bf44b6499eb1441a350dd91"
format_stamp: "Formatted at 2026-09-23 19:29:17 on dist-test-slave-kxfb"
server_key: "0587f84c183498d2b2e4a787e33087cb"
server_key_iv: "e0288f4b5570a0db15ce3d9286d4a48b"
server_key_version: "encryptionkey@0"
I20260923 19:29:17.738121 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:17.753177 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:17.755838 1721 heartbeater.cc:503] Master 127.0.97.62:38587 was elected leader, sending a full tablet report...
I20260923 19:29:17.757867 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:17.758154 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:17.761255 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:17.761338 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:17.761464 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:17.761521 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 19:29:17 dist-test-slave-kxfb krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:17.785802 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.762031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34315 (local address 127.0.97.62:38587)
0923 19:29:17.762216 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:17.762222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:17.763833 (+ 1611us) server_negotiation.cc:407] Connection header received
0923 19:29:17.766268 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:17.766273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:17.766386 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:17.766476 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:17.768853 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.770141 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.772963 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.773554 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.773669 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.778046 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:17.778051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:17.778055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:17.778086 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:17.781247 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.782785 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.782790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.782793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.782936 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.783619 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.783623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.783625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.783944 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:17.784090 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:17.785248 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0923 19:29:17.785479 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 19:29:17.788136 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.762657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38587 (local address 127.0.97.3:34315)
0923 19:29:17.763639 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:17.763689 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:17.765960 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:17.766740 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:17.766752 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:17.767721 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:17.768603 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.768632 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.770408 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.770413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.772807 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.772822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.773804 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.773812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.774154 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.774820 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:17.774842 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:17.777701 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:17.781444 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:17.781458 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:17.781482 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:17.782577 (+ 1095us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:17.783089 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:17.783097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:17.783102 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:17.783447 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:17.784126 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:17.784135 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:17.784721 (+ 586us) client_negotiation.cc:769] Sending connection context
0923 19:29:17.785140 (+ 419us) client_negotiation.cc:240] Negotiation successful
0923 19:29:17.785893 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":387,"threads_started":1}
I20260923 19:29:17.807181 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.794212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44857 (local address 127.0.97.62:38587)
0923 19:29:17.794412 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:17.794417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:17.795371 (+ 954us) server_negotiation.cc:407] Connection header received
0923 19:29:17.795590 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:17.795595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:17.795647 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:17.795722 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:17.796640 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.797338 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.798673 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.799212 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.799320 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.801610 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:17.801616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:17.801621 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:17.801654 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:17.804656 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.805288 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.805294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.805298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.805455 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.805810 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.805815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.805819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.806061 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:17.806181 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:17.806638 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 19:29:17.806835 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 19:29:17.807607 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.794199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38587 (local address 127.0.97.3:44857)
0923 19:29:17.795262 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:17.795286 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:17.795474 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:17.795764 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:17.795768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:17.796209 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:17.796509 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.796518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.797481 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.797486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.798539 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.798550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.799441 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.799448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.799594 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.800310 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:17.800328 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:17.801390 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:17.804755 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:17.804760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:17.804764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:17.805180 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:17.805584 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:17.805588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:17.805591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:17.805724 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:17.806172 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:17.806177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:17.806397 (+ 220us) client_negotiation.cc:769] Sending connection context
0923 19:29:17.806521 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 19:29:17.806876 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":525,"threads_started":1}
I20260923 19:29:17.822198 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45889
I20260923 19:29:17.822220 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45889 every 8 connection(s)
I20260923 19:29:17.823827 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:17.825337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 1864
I20260923 19:29:17.825508 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:17.825950 388 external_mini_cluster.cc:1428] Setting key 2fadd266321eb2f898ce8dadc91aade1
I20260923 19:29:17.826593 1999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38587
I20260923 19:29:17.826921 1999 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:17.827922 1999 heartbeater.cc:511] Master 127.0.97.62:38587 requested a full tablet report, sending...
I20260923 19:29:17.829749 1514 ts_manager.cc:194] Registered new tserver with Master: 553185031bf44b6499eb1441a350dd91 (127.0.97.3:45889)
I20260923 19:29:17.830767 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34315
I20260923 19:29:17.842053 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 19:29:17 dist-test-slave-kxfb krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 19:29:17 dist-test-slave-kxfb krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:17.891206 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.876775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.0.97.62:38587)
0923 19:29:17.877072 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:17.877080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:17.877348 (+ 268us) server_negotiation.cc:407] Connection header received
0923 19:29:17.877509 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:17.877514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:17.877594 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:17.877691 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:17.879095 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.879863 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.881118 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.881567 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.881661 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.885557 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:17.885562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:17.885568 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:17.885613 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:17.888354 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.888973 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.888977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.888980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.889129 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:17.889556 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:17.889562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:17.889566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:17.889920 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:17.890194 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:17.890547 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 19:29:17.890844 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260923 19:29:17.896080 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:29:17.903676 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:17.942333 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.918157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.0.97.2:36075)
0923 19:29:17.932371 (+ 14214us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:17.932380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:17.932407 (+ 27us) server_negotiation.cc:407] Connection header received
0923 19:29:17.932521 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:17.932530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:17.932808 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:17.933035 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:17.934915 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.936370 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.939929 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.941144 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.941278 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.941498 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:17.941779 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 19:29:17.941996 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14036,"thread_start_us":13861,"threads_started":1}
I20260923 19:29:17.942471 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.917937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36075 (local address 127.0.0.1:53794)
0923 19:29:17.931598 (+ 13661us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:17.931613 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:17.931705 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:17.933310 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:17.933316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:17.933346 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:17.934725 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.934743 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.936582 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.936588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.939705 (+ 3117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.939719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.941328 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.941334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.941531 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.941583 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 19:29:17.941701 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 19:29:17.941979 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13524,"mutex_wait_us":161,"thread_start_us":12735,"threads_started":1}
I20260923 19:29:17.943163 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.917871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.0.97.3:45889)
0923 19:29:17.930147 (+ 12276us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:17.930161 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:17.931488 (+ 1327us) server_negotiation.cc:407] Connection header received
0923 19:29:17.932052 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:17.932061 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:17.932352 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:17.932582 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:17.934083 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.935690 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.939904 (+ 4214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.941165 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.941303 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.941564 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:17.942631 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0923 19:29:17.942815 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12004,"thread_start_us":12295,"threads_started":1}
I20260923 19:29:17.943802 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.917492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45889 (local address 127.0.0.1:56696)
0923 19:29:17.931230 (+ 13738us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:17.931315 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:17.931694 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:17.932972 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:17.932982 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:17.933027 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:17.933553 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.933569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.936582 (+ 3013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.936588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.939711 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.939725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.941743 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.941749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.941914 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.941948 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:29:17.942038 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 19:29:17.942195 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13363,"mutex_wait_us":214,"thread_start_us":14388,"threads_started":1}
I20260923 19:29:17.944582 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.932594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33059 (local address 127.0.0.1:40238)
0923 19:29:17.935015 (+ 2421us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:17.935035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:17.935161 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:17.936020 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:17.936025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:17.936049 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:17.936726 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.936737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.938431 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.938436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.940515 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:17.940528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.943742 (+ 3214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.943749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:17.943932 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.943982 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 19:29:17.944080 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 19:29:17.944221 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2274,"mutex_wait_us":35,"thread_start_us":2073,"threads_started":1}
I20260923 19:29:17.944607 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:17.932678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.0.97.1:33059)
0923 19:29:17.934436 (+ 1758us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:17.934446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:17.935290 (+ 844us) server_negotiation.cc:407] Connection header received
0923 19:29:17.935391 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:17.935399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:17.935628 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:17.935817 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:17.937117 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.938266 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.940714 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:17.942160 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:17.943634 (+ 1474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:17.943866 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:17.944140 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 19:29:17.944327 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1539,"thread_start_us":1390,"threads_started":1}
I20260923 19:29:17.946578 1790 tablet_service.cc:1467] Processing CreateTablet for tablet f71132d1b2c24cdba4acbe7f34a3822b (DEFAULT_TABLE table=test-table [id=5b2febe2b8e047ea83a41cd33bf1733f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:17.948016 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71132d1b2c24cdba4acbe7f34a3822b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:17.948060 1929 tablet_service.cc:1467] Processing CreateTablet for tablet f71132d1b2c24cdba4acbe7f34a3822b (DEFAULT_TABLE table=test-table [id=5b2febe2b8e047ea83a41cd33bf1733f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:17.949432 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71132d1b2c24cdba4acbe7f34a3822b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:17.954097 1651 tablet_service.cc:1467] Processing CreateTablet for tablet f71132d1b2c24cdba4acbe7f34a3822b (DEFAULT_TABLE table=test-table [id=5b2febe2b8e047ea83a41cd33bf1733f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:17.955890 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71132d1b2c24cdba4acbe7f34a3822b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:17.961141 2018 tablet_bootstrap.cc:492] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: Bootstrap starting.
I20260923 19:29:17.964449 2018 tablet_bootstrap.cc:654] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:17.965835 2019 tablet_bootstrap.cc:492] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: Bootstrap starting.
I20260923 19:29:17.966405 2018 log.cc:824] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:17.969929 2019 tablet_bootstrap.cc:654] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:17.971223 2019 log.cc:824] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:17.971251 2021 tablet_bootstrap.cc:492] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: Bootstrap starting.
I20260923 19:29:17.972549 2018 tablet_bootstrap.cc:492] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: No bootstrap required, opened a new log
I20260923 19:29:17.972851 2018 ts_tablet_manager.cc:1397] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260923 19:29:17.974537 2019 tablet_bootstrap.cc:492] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: No bootstrap required, opened a new log
I20260923 19:29:17.974848 2019 ts_tablet_manager.cc:1397] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260923 19:29:17.975461 2021 tablet_bootstrap.cc:654] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:17.976562 2021 log.cc:824] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:17.979357 2021 tablet_bootstrap.cc:492] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: No bootstrap required, opened a new log
I20260923 19:29:17.979622 2021 ts_tablet_manager.cc:1397] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 19:29:17.983860 2018 raft_consensus.cc:348] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } }
I20260923 19:29:17.984335 2018 raft_consensus.cc:374] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:17.984443 2018 raft_consensus.cc:729] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd91bad592234245bcfe50254d56b4ce, State: Initialized, Role: FOLLOWER
I20260923 19:29:17.985119 2018 consensus_queue.cc:260] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } }
I20260923 19:29:17.985545 2019 raft_consensus.cc:348] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } }
I20260923 19:29:17.986065 2019 raft_consensus.cc:374] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:17.986166 2019 raft_consensus.cc:729] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553185031bf44b6499eb1441a350dd91, State: Initialized, Role: FOLLOWER
I20260923 19:29:17.986696 1860 heartbeater.cc:503] Master 127.0.97.62:38587 was elected leader, sending a full tablet report...
I20260923 19:29:17.987035 2018 ts_tablet_manager.cc:1428] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260923 19:29:17.987439 2019 consensus_queue.cc:260] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } }
I20260923 19:29:17.988443 1999 heartbeater.cc:503] Master 127.0.97.62:38587 was elected leader, sending a full tablet report...
I20260923 19:29:17.988538 2021 raft_consensus.cc:348] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } }
I20260923 19:29:17.988994 2019 ts_tablet_manager.cc:1428] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260923 19:29:17.989008 2021 raft_consensus.cc:374] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:17.989099 2021 raft_consensus.cc:729] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9bad2d029f4eecad0445cfba260bb5, State: Initialized, Role: FOLLOWER
I20260923 19:29:17.989914 2021 consensus_queue.cc:260] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } }
I20260923 19:29:17.992408 2021 ts_tablet_manager.cc:1428] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260923 19:29:17.996698 1722 tablet.cc:2369] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:29:18.012334 1861 tablet.cc:2369] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:29:18.075834 2000 tablet.cc:2369] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:29:18.298955 2025 raft_consensus.cc:482] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:18.299288 2025 raft_consensus.cc:504] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } }
I20260923 19:29:18.300947 2025 leader_election.cc:288] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f9bad2d029f4eecad0445cfba260bb5 (127.0.97.1:33059), cd91bad592234245bcfe50254d56b4ce (127.0.97.2:36075)
I20260923 19:29:18.310074 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.301526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60721 (local address 127.0.97.2:36075)
0923 19:29:18.301731 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:18.301737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:18.302369 (+ 632us) server_negotiation.cc:407] Connection header received
0923 19:29:18.302564 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:18.302570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:18.302673 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:18.302786 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:18.303519 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.304691 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.307610 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.308920 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.309071 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.309200 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:18.309618 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 19:29:18.309769 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 19:29:18.310066 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.301533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49851 (local address 127.0.97.1:33059)
0923 19:29:18.301731 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:18.301737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:18.302340 (+ 603us) server_negotiation.cc:407] Connection header received
0923 19:29:18.302564 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:18.302570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:18.302673 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:18.302786 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:18.303519 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.304688 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.307607 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.308922 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.309071 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.309200 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:18.309618 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 19:29:18.309769 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 19:29:18.310238 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.301353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36075 (local address 127.0.97.3:60721)
0923 19:29:18.302171 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:18.302196 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:18.302403 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:18.302815 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:18.302820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:18.302865 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:18.303329 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.303338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.304874 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.304880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.307409 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.307425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.309149 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.309154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.309382 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.309459 (+ 77us) client_negotiation.cc:769] Sending connection context
0923 19:29:18.309564 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 19:29:18.309776 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":320,"spinlock_wait_cycles":36224,"thread_start_us":530,"threads_started":1}
I20260923 19:29:18.310720 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.301417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33059 (local address 127.0.97.3:49851)
0923 19:29:18.302196 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:18.302210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:18.302393 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:18.302815 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:18.302820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:18.302863 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:18.303320 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.303333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.304877 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.304882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.307409 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.307425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.309123 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.309130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.309388 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.309459 (+ 71us) client_negotiation.cc:769] Sending connection context
0923 19:29:18.309564 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 19:29:18.309776 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":274,"spinlock_wait_cycles":14976,"thread_start_us":460,"threads_started":1}
I20260923 19:29:18.311671 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71132d1b2c24cdba4acbe7f34a3822b" candidate_uuid: "553185031bf44b6499eb1441a350dd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd91bad592234245bcfe50254d56b4ce" is_pre_election: true
I20260923 19:29:18.312146 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71132d1b2c24cdba4acbe7f34a3822b" candidate_uuid: "553185031bf44b6499eb1441a350dd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f9bad2d029f4eecad0445cfba260bb5" is_pre_election: true
I20260923 19:29:18.312711 1810 raft_consensus.cc:2454] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 553185031bf44b6499eb1441a350dd91 in term 0.
I20260923 19:29:18.312740 1676 raft_consensus.cc:2454] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 553185031bf44b6499eb1441a350dd91 in term 0.
I20260923 19:29:18.313445 1882 leader_election.cc:302] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f9bad2d029f4eecad0445cfba260bb5, 553185031bf44b6499eb1441a350dd91; no voters:
I20260923 19:29:18.313846 2025 raft_consensus.cc:2781] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:18.314008 2025 raft_consensus.cc:482] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:18.314101 2025 raft_consensus.cc:3037] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:18.316207 2025 raft_consensus.cc:504] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } }
I20260923 19:29:18.316751 2025 leader_election.cc:288] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [CANDIDATE]: Term 1 election: Requested vote from peers 3f9bad2d029f4eecad0445cfba260bb5 (127.0.97.1:33059), cd91bad592234245bcfe50254d56b4ce (127.0.97.2:36075)
I20260923 19:29:18.317369 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71132d1b2c24cdba4acbe7f34a3822b" candidate_uuid: "553185031bf44b6499eb1441a350dd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd91bad592234245bcfe50254d56b4ce"
I20260923 19:29:18.317402 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71132d1b2c24cdba4acbe7f34a3822b" candidate_uuid: "553185031bf44b6499eb1441a350dd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f9bad2d029f4eecad0445cfba260bb5"
I20260923 19:29:18.317535 1810 raft_consensus.cc:3037] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:18.317653 1676 raft_consensus.cc:3037] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:18.319727 1810 raft_consensus.cc:2454] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553185031bf44b6499eb1441a350dd91 in term 1.
I20260923 19:29:18.320163 1880 leader_election.cc:302] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 553185031bf44b6499eb1441a350dd91, cd91bad592234245bcfe50254d56b4ce; no voters:
I20260923 19:29:18.320431 2025 raft_consensus.cc:2781] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:18.320454 1676 raft_consensus.cc:2454] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553185031bf44b6499eb1441a350dd91 in term 1.
I20260923 19:29:18.320894 2025 raft_consensus.cc:686] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 1 LEADER]: Becoming Leader. State: Replica: 553185031bf44b6499eb1441a350dd91, State: Running, Role: LEADER
I20260923 19:29:18.321453 2025 consensus_queue.cc:237] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } }
I20260923 19:29:18.326069 1514 catalog_manager.cc:5495] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 553185031bf44b6499eb1441a350dd91 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "553185031bf44b6499eb1441a350dd91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } health_report { overall_health: UNKNOWN } } }
I20260923 19:29:18.422168 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.415859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.0.97.3:45889)
0923 19:29:18.416100 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:18.416105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:18.416652 (+ 547us) server_negotiation.cc:407] Connection header received
0923 19:29:18.416947 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:18.416951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:18.417030 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:18.417207 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:18.417655 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.418434 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.420009 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.420426 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.420519 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.421006 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:18.421578 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:18.421695 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:18.421861 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 19:29:18.421928 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 19:29:18.431411 1810 raft_consensus.cc:1261] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Refusing update from remote peer 553185031bf44b6499eb1441a350dd91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:18.431560 1676 raft_consensus.cc:1261] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Refusing update from remote peer 553185031bf44b6499eb1441a350dd91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:18.432439 2029 consensus_queue.cc:1035] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:18.432832 2025 consensus_queue.cc:1035] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:18.465711 1514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:18.466022 1514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:18.470044 1514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275: Sending DeleteTablet for 3 replicas of tablet f71132d1b2c24cdba4acbe7f34a3822b
I20260923 19:29:18.471603 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet f71132d1b2c24cdba4acbe7f34a3822b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53794
I20260923 19:29:18.471560 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet f71132d1b2c24cdba4acbe7f34a3822b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40238
I20260923 19:29:18.471709 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet f71132d1b2c24cdba4acbe7f34a3822b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56696
I20260923 19:29:18.476861 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:29:18.477779 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:18.484939 2044 tablet_replica.cc:330] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: stopping tablet replica
I20260923 19:29:18.485489 2044 raft_consensus.cc:2229] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:29:18.486200 2044 raft_consensus.cc:2258] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:18.487231 1650 tablet_service.cc:1467] Processing CreateTablet for tablet e0404896b28f4cb6b1314d0d12a7e2dd (DEFAULT_TABLE table=test-table [id=a8acfd8c541a4b618586882339d75a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:18.487844 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0404896b28f4cb6b1314d0d12a7e2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:18.487941 1929 tablet_service.cc:1467] Processing CreateTablet for tablet e0404896b28f4cb6b1314d0d12a7e2dd (DEFAULT_TABLE table=test-table [id=a8acfd8c541a4b618586882339d75a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:18.488602 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0404896b28f4cb6b1314d0d12a7e2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:18.488687 2044 ts_tablet_manager.cc:1905] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:18.490842 1789 tablet_service.cc:1467] Processing CreateTablet for tablet e0404896b28f4cb6b1314d0d12a7e2dd (DEFAULT_TABLE table=test-table [id=a8acfd8c541a4b618586882339d75a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:18.491329 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0404896b28f4cb6b1314d0d12a7e2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:18.493377 2043 tablet_replica.cc:330] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: stopping tablet replica
I20260923 19:29:18.493992 2043 raft_consensus.cc:2229] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:18.494611 2043 raft_consensus.cc:2258] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:18.496311 2042 tablet_replica.cc:330] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: stopping tablet replica
I20260923 19:29:18.496728 2044 ts_tablet_manager.cc:1918] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:29:18.496908 2044 log.cc:1196] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/f71132d1b2c24cdba4acbe7f34a3822b
I20260923 19:29:18.497431 2044 ts_tablet_manager.cc:1939] T f71132d1b2c24cdba4acbe7f34a3822b P 553185031bf44b6499eb1441a350dd91: Deleting consensus metadata
I20260923 19:29:18.498389 2043 ts_tablet_manager.cc:1905] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:18.499374 1500 catalog_manager.cc:4841] TS 553185031bf44b6499eb1441a350dd91 (127.0.97.3:45889): tablet f71132d1b2c24cdba4acbe7f34a3822b (table test-table [id=5b2febe2b8e047ea83a41cd33bf1733f]) successfully deleted
I20260923 19:29:18.500056 2042 raft_consensus.cc:2229] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:18.506338 2046 tablet_bootstrap.cc:492] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: Bootstrap starting.
I20260923 19:29:18.507072 2042 raft_consensus.cc:2258] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:18.508669 2046 tablet_bootstrap.cc:654] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:18.509730 2042 ts_tablet_manager.cc:1905] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:18.513090 2045 tablet_bootstrap.cc:492] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: Bootstrap starting.
I20260923 19:29:18.513813 2047 tablet_bootstrap.cc:492] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: Bootstrap starting.
I20260923 19:29:18.514853 2045 tablet_bootstrap.cc:654] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:18.518252 2046 tablet_bootstrap.cc:492] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: No bootstrap required, opened a new log
I20260923 19:29:18.518435 2046 ts_tablet_manager.cc:1397] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20260923 19:29:18.518677 2043 ts_tablet_manager.cc:1918] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:29:18.518859 2043 log.cc:1196] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/f71132d1b2c24cdba4acbe7f34a3822b
I20260923 19:29:18.519402 2043 ts_tablet_manager.cc:1939] T f71132d1b2c24cdba4acbe7f34a3822b P 3f9bad2d029f4eecad0445cfba260bb5: Deleting consensus metadata
I20260923 19:29:18.519304 2046 raft_consensus.cc:348] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.519570 2046 raft_consensus.cc:374] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:18.519666 2046 raft_consensus.cc:729] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9bad2d029f4eecad0445cfba260bb5, State: Initialized, Role: FOLLOWER
I20260923 19:29:18.519999 2046 consensus_queue.cc:260] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.521649 1500 catalog_manager.cc:4841] TS 3f9bad2d029f4eecad0445cfba260bb5 (127.0.97.1:33059): tablet f71132d1b2c24cdba4acbe7f34a3822b (table test-table [id=5b2febe2b8e047ea83a41cd33bf1733f]) successfully deleted
I20260923 19:29:18.522270 2047 tablet_bootstrap.cc:654] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:18.523030 2046 ts_tablet_manager.cc:1428] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 19:29:18.524044 2042 ts_tablet_manager.cc:1918] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:29:18.524206 2042 log.cc:1196] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/f71132d1b2c24cdba4acbe7f34a3822b
I20260923 19:29:18.524621 2042 ts_tablet_manager.cc:1939] T f71132d1b2c24cdba4acbe7f34a3822b P cd91bad592234245bcfe50254d56b4ce: Deleting consensus metadata
I20260923 19:29:18.526019 2045 tablet_bootstrap.cc:492] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: No bootstrap required, opened a new log
I20260923 19:29:18.526221 2045 ts_tablet_manager.cc:1397] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260923 19:29:18.526592 2047 tablet_bootstrap.cc:492] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: No bootstrap required, opened a new log
I20260923 19:29:18.526597 1498 catalog_manager.cc:4841] TS cd91bad592234245bcfe50254d56b4ce (127.0.97.2:36075): tablet f71132d1b2c24cdba4acbe7f34a3822b (table test-table [id=5b2febe2b8e047ea83a41cd33bf1733f]) successfully deleted
I20260923 19:29:18.526763 2047 ts_tablet_manager.cc:1397] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20260923 19:29:18.527585 2045 raft_consensus.cc:348] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.527792 2045 raft_consensus.cc:374] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:18.527879 2045 raft_consensus.cc:729] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd91bad592234245bcfe50254d56b4ce, State: Initialized, Role: FOLLOWER
I20260923 19:29:18.527916 2047 raft_consensus.cc:348] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.528116 2045 consensus_queue.cc:260] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.528281 2047 raft_consensus.cc:374] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:18.528535 2047 raft_consensus.cc:729] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553185031bf44b6499eb1441a350dd91, State: Initialized, Role: FOLLOWER
I20260923 19:29:18.528630 2045 ts_tablet_manager.cc:1428] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 19:29:18.528825 2047 consensus_queue.cc:260] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.529444 2047 ts_tablet_manager.cc:1428] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 19:29:18.585278 2025 raft_consensus.cc:482] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:18.585536 2025 raft_consensus.cc:504] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.586315 2025 leader_election.cc:288] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f9bad2d029f4eecad0445cfba260bb5 (127.0.97.1:33059), cd91bad592234245bcfe50254d56b4ce (127.0.97.2:36075)
I20260923 19:29:18.587054 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0404896b28f4cb6b1314d0d12a7e2dd" candidate_uuid: "553185031bf44b6499eb1441a350dd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f9bad2d029f4eecad0445cfba260bb5" is_pre_election: true
I20260923 19:29:18.587088 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0404896b28f4cb6b1314d0d12a7e2dd" candidate_uuid: "553185031bf44b6499eb1441a350dd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd91bad592234245bcfe50254d56b4ce" is_pre_election: true
I20260923 19:29:18.587332 1676 raft_consensus.cc:2454] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 553185031bf44b6499eb1441a350dd91 in term 0.
I20260923 19:29:18.587394 1810 raft_consensus.cc:2454] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 553185031bf44b6499eb1441a350dd91 in term 0.
I20260923 19:29:18.587898 1882 leader_election.cc:302] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f9bad2d029f4eecad0445cfba260bb5, 553185031bf44b6499eb1441a350dd91, cd91bad592234245bcfe50254d56b4ce; no voters:
I20260923 19:29:18.588192 2025 raft_consensus.cc:2781] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:18.588341 2025 raft_consensus.cc:482] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:18.588423 2025 raft_consensus.cc:3037] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:18.591305 2025 raft_consensus.cc:504] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.591991 2025 leader_election.cc:288] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [CANDIDATE]: Term 1 election: Requested vote from peers 3f9bad2d029f4eecad0445cfba260bb5 (127.0.97.1:33059), cd91bad592234245bcfe50254d56b4ce (127.0.97.2:36075)
I20260923 19:29:18.592551 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0404896b28f4cb6b1314d0d12a7e2dd" candidate_uuid: "553185031bf44b6499eb1441a350dd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f9bad2d029f4eecad0445cfba260bb5"
I20260923 19:29:18.592640 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0404896b28f4cb6b1314d0d12a7e2dd" candidate_uuid: "553185031bf44b6499eb1441a350dd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd91bad592234245bcfe50254d56b4ce"
I20260923 19:29:18.592823 1676 raft_consensus.cc:3037] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:18.592849 1810 raft_consensus.cc:3037] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:18.595007 1810 raft_consensus.cc:2454] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553185031bf44b6499eb1441a350dd91 in term 1.
I20260923 19:29:18.595407 1880 leader_election.cc:302] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 553185031bf44b6499eb1441a350dd91, cd91bad592234245bcfe50254d56b4ce; no voters:
I20260923 19:29:18.595734 2025 raft_consensus.cc:2781] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:18.595886 2025 raft_consensus.cc:686] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 1 LEADER]: Becoming Leader. State: Replica: 553185031bf44b6499eb1441a350dd91, State: Running, Role: LEADER
I20260923 19:29:18.596269 1676 raft_consensus.cc:2454] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553185031bf44b6499eb1441a350dd91 in term 1.
I20260923 19:29:18.596223 2025 consensus_queue.cc:237] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.600379 1514 catalog_manager.cc:5495] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 553185031bf44b6499eb1441a350dd91 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "553185031bf44b6499eb1441a350dd91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } health_report { overall_health: HEALTHY } } }
Sep 23 19:29:18 dist-test-slave-kxfb krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:18.661731 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.645207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.0.97.62:38587)
0923 19:29:18.646084 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:18.646089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:18.646419 (+ 330us) server_negotiation.cc:407] Connection header received
0923 19:29:18.646535 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:18.646539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:18.646612 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:18.646722 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:18.648300 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.649145 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.650996 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.651450 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.651550 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.655742 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:18.655747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:18.655751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:18.655785 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:18.658661 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:18.659547 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:18.659551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:18.659554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:18.659684 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:18.660095 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:18.660098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:18.660100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:18.660376 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:18.660505 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:18.661031 (+ 526us) server_negotiation.cc:299] Negotiation successful
0923 19:29:18.661355 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":736,"thread_start_us":725,"threads_started":1}
I20260923 19:29:18.661762 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.645204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38587 (local address 127.0.0.1:46070)
0923 19:29:18.646152 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:18.646176 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:18.646391 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:18.646818 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:18.646823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:18.647758 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:18.648092 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.648104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.649303 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.649308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.650842 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.650856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.651629 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.651636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.651825 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.652657 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:18.652681 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:18.655504 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:18.658804 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:18.658817 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:18.658825 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:18.659411 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:18.659794 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:18.659799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:18.659802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:18.659982 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:18.660520 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:18.660529 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:18.660744 (+ 215us) client_negotiation.cc:769] Sending connection context
0923 19:29:18.660930 (+ 186us) client_negotiation.cc:240] Negotiation successful
0923 19:29:18.661237 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":482,"threads_started":1}
I20260923 19:29:18.669360 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46070:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 19:29:18.670825 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:18.678753 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 6d47c7c199b54b7dbe4dcfcb922e6c91 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f5622d10263945819d1cacbe77068359]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 19:29:18.679121 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 6d47c7c199b54b7dbe4dcfcb922e6c91 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f5622d10263945819d1cacbe77068359]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 19:29:18.679134 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 6d47c7c199b54b7dbe4dcfcb922e6c91 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f5622d10263945819d1cacbe77068359]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 19:29:18.679376 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d47c7c199b54b7dbe4dcfcb922e6c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:18.679703 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d47c7c199b54b7dbe4dcfcb922e6c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:18.679736 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d47c7c199b54b7dbe4dcfcb922e6c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:18.687592 2045 tablet_bootstrap.cc:492] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: Bootstrap starting.
I20260923 19:29:18.688506 2046 tablet_bootstrap.cc:492] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: Bootstrap starting.
I20260923 19:29:18.689399 2045 tablet_bootstrap.cc:654] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:18.691128 2046 tablet_bootstrap.cc:654] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:18.693125 2047 tablet_bootstrap.cc:492] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: Bootstrap starting.
I20260923 19:29:18.694051 2046 tablet_bootstrap.cc:492] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: No bootstrap required, opened a new log
I20260923 19:29:18.694222 2046 ts_tablet_manager.cc:1397] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260923 19:29:18.694988 2047 tablet_bootstrap.cc:654] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:18.695006 2046 raft_consensus.cc:348] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.695279 2046 raft_consensus.cc:374] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:18.695343 2046 raft_consensus.cc:729] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9bad2d029f4eecad0445cfba260bb5, State: Initialized, Role: FOLLOWER
I20260923 19:29:18.695587 2046 consensus_queue.cc:260] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.696100 2046 ts_tablet_manager.cc:1428] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 19:29:18.696502 2026 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.696825 2026 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:18.697199 2045 tablet_bootstrap.cc:492] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: No bootstrap required, opened a new log
I20260923 19:29:18.697398 2045 ts_tablet_manager.cc:1397] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260923 19:29:18.699551 2045 raft_consensus.cc:348] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.699712 2047 tablet_bootstrap.cc:492] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: No bootstrap required, opened a new log
I20260923 19:29:18.699798 2045 raft_consensus.cc:374] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:18.699872 2047 ts_tablet_manager.cc:1397] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260923 19:29:18.699877 2045 raft_consensus.cc:729] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd91bad592234245bcfe50254d56b4ce, State: Initialized, Role: FOLLOWER
I20260923 19:29:18.700205 2045 consensus_queue.cc:260] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.700623 2047 raft_consensus.cc:348] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.700883 2047 raft_consensus.cc:374] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:18.700696 2024 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.700948 2047 raft_consensus.cc:729] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553185031bf44b6499eb1441a350dd91, State: Initialized, Role: FOLLOWER
I20260923 19:29:18.701045 2024 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:18.701673 2047 consensus_queue.cc:260] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.701985 2045 ts_tablet_manager.cc:1428] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260923 19:29:18.702077 2025 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.702387 2025 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:18.703223 2047 ts_tablet_manager.cc:1428] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260923 19:29:18.783628 2024 raft_consensus.cc:482] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:18.783926 2024 raft_consensus.cc:504] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.785785 2024 leader_election.cc:288] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f9bad2d029f4eecad0445cfba260bb5 (127.0.97.1:33059), 553185031bf44b6499eb1441a350dd91 (127.0.97.3:45889)
I20260923 19:29:18.794442 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.786143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33059 (local address 127.0.97.2:55315)
0923 19:29:18.786851 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:18.786869 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:18.787125 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:18.787734 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:18.787739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:18.787765 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:18.788204 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.788217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.789913 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.789918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.791771 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.791784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.793621 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.793628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.793792 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.793826 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:29:18.793913 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 19:29:18.794053 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":437,"threads_started":1}
I20260923 19:29:18.794407 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.786205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55315 (local address 127.0.97.1:33059)
0923 19:29:18.786415 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:18.786420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:18.786989 (+ 569us) server_negotiation.cc:407] Connection header received
0923 19:29:18.787333 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:18.787339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:18.787421 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:18.787494 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:18.788401 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.789617 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.791959 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.793220 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.793333 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.793452 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:18.793960 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 19:29:18.794094 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 19:29:18.795544 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d47c7c199b54b7dbe4dcfcb922e6c91" candidate_uuid: "cd91bad592234245bcfe50254d56b4ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f9bad2d029f4eecad0445cfba260bb5" is_pre_election: true
I20260923 19:29:18.795820 1676 raft_consensus.cc:2454] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd91bad592234245bcfe50254d56b4ce in term 0.
I20260923 19:29:18.796294 1743 leader_election.cc:302] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f9bad2d029f4eecad0445cfba260bb5, cd91bad592234245bcfe50254d56b4ce; no voters:
I20260923 19:29:18.796697 2024 raft_consensus.cc:2781] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:18.796805 2024 raft_consensus.cc:482] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:18.796914 2024 raft_consensus.cc:3037] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:18.796980 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.787214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57297 (local address 127.0.97.3:45889)
0923 19:29:18.787420 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:18.787430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:18.788171 (+ 741us) server_negotiation.cc:407] Connection header received
0923 19:29:18.788387 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:18.788394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:18.788482 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:18.788635 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:18.790256 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.791563 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.793883 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.795189 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.795336 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.795444 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:18.796097 (+ 653us) server_negotiation.cc:299] Negotiation successful
0923 19:29:18.796241 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 19:29:18.797857 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.787044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45889 (local address 127.0.97.2:57297)
0923 19:29:18.788033 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:18.788052 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:18.788231 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:18.788649 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:18.788653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:18.788678 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:18.790031 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.790043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.791741 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.791747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.793687 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.793698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.795647 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.795656 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.795888 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.795928 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 19:29:18.797211 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0923 19:29:18.797396 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":1,"thread_start_us":552,"threads_started":1}
I20260923 19:29:18.798915 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d47c7c199b54b7dbe4dcfcb922e6c91" candidate_uuid: "cd91bad592234245bcfe50254d56b4ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553185031bf44b6499eb1441a350dd91" is_pre_election: true
I20260923 19:29:18.799401 1954 raft_consensus.cc:2454] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd91bad592234245bcfe50254d56b4ce in term 0.
I20260923 19:29:18.799630 2024 raft_consensus.cc:504] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.800221 2024 leader_election.cc:288] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [CANDIDATE]: Term 1 election: Requested vote from peers 3f9bad2d029f4eecad0445cfba260bb5 (127.0.97.1:33059), 553185031bf44b6499eb1441a350dd91 (127.0.97.3:45889)
I20260923 19:29:18.800844 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d47c7c199b54b7dbe4dcfcb922e6c91" candidate_uuid: "cd91bad592234245bcfe50254d56b4ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f9bad2d029f4eecad0445cfba260bb5"
I20260923 19:29:18.801088 1676 raft_consensus.cc:3037] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:18.801051 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d47c7c199b54b7dbe4dcfcb922e6c91" candidate_uuid: "cd91bad592234245bcfe50254d56b4ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553185031bf44b6499eb1441a350dd91"
I20260923 19:29:18.801292 1954 raft_consensus.cc:3037] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:18.804126 1676 raft_consensus.cc:2454] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd91bad592234245bcfe50254d56b4ce in term 1.
I20260923 19:29:18.804601 1743 leader_election.cc:302] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f9bad2d029f4eecad0445cfba260bb5, cd91bad592234245bcfe50254d56b4ce; no voters:
I20260923 19:29:18.804912 1954 raft_consensus.cc:2454] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd91bad592234245bcfe50254d56b4ce in term 1.
I20260923 19:29:18.805260 2024 raft_consensus.cc:2781] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:18.805828 2024 raft_consensus.cc:686] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [term 1 LEADER]: Becoming Leader. State: Replica: cd91bad592234245bcfe50254d56b4ce, State: Running, Role: LEADER
I20260923 19:29:18.806509 2024 consensus_queue.cc:237] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } }
I20260923 19:29:18.811555 2064 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: TxnStatusTablet state changed. Reason: New leader cd91bad592234245bcfe50254d56b4ce. Latest consensus state: current_term: 1 leader_uuid: "cd91bad592234245bcfe50254d56b4ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.811944 2064 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: This TxnStatusTablet replica's current role is: LEADER
I20260923 19:29:18.813082 2066 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 19:29:18.813371 2066 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 19:29:18.813803 1511 catalog_manager.cc:5495] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce reported cstate change: term changed from 0 to 1, leader changed from <none> to cd91bad592234245bcfe50254d56b4ce (127.0.97.2). New cstate: current_term: 1 leader_uuid: "cd91bad592234245bcfe50254d56b4ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } health_report { overall_health: UNKNOWN } } }
I20260923 19:29:18.876188 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.866177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36075 (local address 127.0.0.1:42020)
0923 19:29:18.866472 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:18.866496 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:18.866654 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:18.868088 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:18.868094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:18.868114 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:29:18.868386 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.868399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.870866 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.870872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.872592 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.872609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.873719 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.873726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.873981 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.874079 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:29:18.875471 (+ 1392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:29:18.875496 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 19:29:18.875610 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 19:29:18.875804 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 19:29:18.876207 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.866373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.0.97.2:36075)
0923 19:29:18.867528 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:18.867535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:18.867560 (+ 25us) server_negotiation.cc:407] Connection header received
0923 19:29:18.867669 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:18.867674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:18.867794 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:18.867967 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:18.868624 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.870668 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.872809 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.873518 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.873674 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.874432 (+ 758us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:18.875331 (+ 899us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:18.875502 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:18.875665 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 19:29:18.875764 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":802,"threads_started":1}
I20260923 19:29:18.882081 1676 raft_consensus.cc:1261] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Refusing update from remote peer cd91bad592234245bcfe50254d56b4ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:18.882468 1954 raft_consensus.cc:1261] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91 [term 1 FOLLOWER]: Refusing update from remote peer cd91bad592234245bcfe50254d56b4ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:18.883020 2024 consensus_queue.cc:1035] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:18.883347 2064 consensus_queue.cc:1035] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:18.890843 2026 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: TxnStatusTablet state changed. Reason: New leader cd91bad592234245bcfe50254d56b4ce. Latest consensus state: current_term: 1 leader_uuid: "cd91bad592234245bcfe50254d56b4ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.891172 2026 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:18.892999 2024 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd91bad592234245bcfe50254d56b4ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.893324 2024 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: This TxnStatusTablet replica's current role is: LEADER
I20260923 19:29:18.900902 2026 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd91bad592234245bcfe50254d56b4ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.901149 2026 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 3f9bad2d029f4eecad0445cfba260bb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:18.904134 2025 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: TxnStatusTablet state changed. Reason: New leader cd91bad592234245bcfe50254d56b4ce. Latest consensus state: current_term: 1 leader_uuid: "cd91bad592234245bcfe50254d56b4ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.904436 2025 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:18.907191 2064 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd91bad592234245bcfe50254d56b4ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.907542 2064 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P cd91bad592234245bcfe50254d56b4ce: This TxnStatusTablet replica's current role is: LEADER
I20260923 19:29:18.909981 2033 mvcc.cc:205] Tried to move back new op lower bound from 7332625444372201472 to 7332625444075626496. Current Snapshot: MvccSnapshot[applied={T|T < 7332625444372201472}]
I20260923 19:29:18.910459 2025 tablet_replica.cc:439] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd91bad592234245bcfe50254d56b4ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 } } peers { permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 } } peers { permanent_uuid: "553185031bf44b6499eb1441a350dd91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45889 } } }
I20260923 19:29:18.910723 2025 tablet_replica.cc:442] T 6d47c7c199b54b7dbe4dcfcb922e6c91 P 553185031bf44b6499eb1441a350dd91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:18.944788 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.934764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53229 (local address 127.0.97.2:36075)
0923 19:29:18.935003 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:18.935011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:18.939185 (+ 4174us) server_negotiation.cc:407] Connection header received
0923 19:29:18.939402 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:18.939407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:18.939484 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:18.939562 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:18.940165 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.940977 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.942842 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.943378 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.943495 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.944003 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:18.944126 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:18.944200 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:18.944328 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 19:29:18.944413 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 19:29:18.945196 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:18.934652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36075 (local address 127.0.97.3:53229)
0923 19:29:18.939041 (+ 4389us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:18.939063 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:18.939256 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:18.939602 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:18.939606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:18.939618 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:29:18.940012 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.940022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.941111 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.941116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.942708 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:18.942720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:18.943566 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:18.943573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:18.943717 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:18.943818 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:29:18.944217 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:29:18.944229 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 19:29:18.944305 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 19:29:18.944473 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4180,"thread_start_us":3913,"threads_started":1}
I20260923 19:29:18.957820 1810 raft_consensus.cc:1261] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Refusing update from remote peer 553185031bf44b6499eb1441a350dd91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:18.958770 2029 consensus_queue.cc:1035] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd91bad592234245bcfe50254d56b4ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:18.958993 1676 raft_consensus.cc:1261] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Refusing update from remote peer 553185031bf44b6499eb1441a350dd91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:18.959679 2029 consensus_queue.cc:1035] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f9bad2d029f4eecad0445cfba260bb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:19.012501 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:19.005370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49757 (local address 127.0.97.3:45889)
0923 19:29:19.005553 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:19.005558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:19.006366 (+ 808us) server_negotiation.cc:407] Connection header received
0923 19:29:19.006703 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:19.006709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:19.006805 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:19.006930 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:19.007769 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.008520 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.010480 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.010975 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.011116 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:19.011770 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:19.011889 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:19.011961 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:19.012184 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 19:29:19.012261 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 19:29:19.012841 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:19.005414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45889 (local address 127.0.97.2:49757)
0923 19:29:19.006224 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:19.006245 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:19.006504 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:19.007126 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:19.007132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:19.007146 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:29:19.007585 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:19.007599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.008651 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.008660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:19.010173 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:19.010184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.011217 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.011225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:19.011472 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:19.011568 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:29:19.012051 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:29:19.012066 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 19:29:19.012160 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 19:29:19.012340 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":412,"threads_started":1}
I20260923 19:29:19.071666 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:19.071967 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:19.075534 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eab7a2276ee4d358242e075c50cc275: Sending DeleteTablet for 3 replicas of tablet e0404896b28f4cb6b1314d0d12a7e2dd
I20260923 19:29:19.076473 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet e0404896b28f4cb6b1314d0d12a7e2dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53794
I20260923 19:29:19.076521 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet e0404896b28f4cb6b1314d0d12a7e2dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40238
I20260923 19:29:19.076550 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet e0404896b28f4cb6b1314d0d12a7e2dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56696
I20260923 19:29:19.093138 2082 tablet_replica.cc:330] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: stopping tablet replica
I20260923 19:29:19.093478 2082 raft_consensus.cc:2229] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:19.093843 2082 raft_consensus.cc:2258] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:19.094753 2083 tablet_replica.cc:330] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: stopping tablet replica
I20260923 19:29:19.095233 2082 ts_tablet_manager.cc:1905] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:19.095521 2083 raft_consensus.cc:2229] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:19.096174 2083 raft_consensus.cc:2258] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:19.097812 2083 ts_tablet_manager.cc:1905] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:19.098716 2084 tablet_replica.cc:330] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: stopping tablet replica
I20260923 19:29:19.100945 2084 raft_consensus.cc:2229] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:29:19.101466 2084 raft_consensus.cc:2258] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 19:29:19 dist-test-slave-kxfb krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:19.103456 2082 ts_tablet_manager.cc:1918] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:29:19.103607 2082 log.cc:1196] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/e0404896b28f4cb6b1314d0d12a7e2dd
I20260923 19:29:19.104103 2082 ts_tablet_manager.cc:1939] T e0404896b28f4cb6b1314d0d12a7e2dd P cd91bad592234245bcfe50254d56b4ce: Deleting consensus metadata
I20260923 19:29:19.104331 2084 ts_tablet_manager.cc:1905] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:19.106025 1498 catalog_manager.cc:4841] TS cd91bad592234245bcfe50254d56b4ce (127.0.97.2:36075): tablet e0404896b28f4cb6b1314d0d12a7e2dd (table test-table [id=a8acfd8c541a4b618586882339d75a29]) successfully deleted
I20260923 19:29:19.111629 2084 ts_tablet_manager.cc:1918] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:29:19.111783 2084 log.cc:1196] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/e0404896b28f4cb6b1314d0d12a7e2dd
I20260923 19:29:19.112200 2084 ts_tablet_manager.cc:1939] T e0404896b28f4cb6b1314d0d12a7e2dd P 553185031bf44b6499eb1441a350dd91: Deleting consensus metadata
I20260923 19:29:19.113003 2083 ts_tablet_manager.cc:1918] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:29:19.113204 2083 log.cc:1196] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/e0404896b28f4cb6b1314d0d12a7e2dd
I20260923 19:29:19.113554 1500 catalog_manager.cc:4841] TS 553185031bf44b6499eb1441a350dd91 (127.0.97.3:45889): tablet e0404896b28f4cb6b1314d0d12a7e2dd (table test-table [id=a8acfd8c541a4b618586882339d75a29]) successfully deleted
I20260923 19:29:19.113732 2083 ts_tablet_manager.cc:1939] T e0404896b28f4cb6b1314d0d12a7e2dd P 3f9bad2d029f4eecad0445cfba260bb5: Deleting consensus metadata
I20260923 19:29:19.114221 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:19.087808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.0.97.1:33059)
0923 19:29:19.087997 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:19.088003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:19.092526 (+ 4523us) server_negotiation.cc:407] Connection header received
0923 19:29:19.092741 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:19.092748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:19.092825 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:19.092884 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:19.094728 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.095887 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.097651 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.098313 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.098410 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:19.104008 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:19.104025 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:19.104047 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:19.104096 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:19.110893 (+ 6797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.111770 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.111779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.111783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.111928 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.112338 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.112342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.112346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.112735 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:19.113032 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:19.113559 (+ 527us) server_negotiation.cc:299] Negotiation successful
0923 19:29:19.113820 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 19:29:19.114796 1500 catalog_manager.cc:4841] TS 3f9bad2d029f4eecad0445cfba260bb5 (127.0.97.1:33059): tablet e0404896b28f4cb6b1314d0d12a7e2dd (table test-table [id=a8acfd8c541a4b618586882339d75a29]) successfully deleted
W20260923 19:29:19.115712 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49724
I20260923 19:29:19.132746 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:19.118819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.0.97.62:38587)
0923 19:29:19.119015 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:19.119020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:19.119255 (+ 235us) server_negotiation.cc:407] Connection header received
0923 19:29:19.119410 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:19.119415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:19.119497 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:19.119608 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:19.120901 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.121889 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.123321 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.123932 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.124092 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:19.126458 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:19.126465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:19.126470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:19.126510 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:19.130008 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.130747 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.130753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.130756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.130899 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.131250 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.131255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.131259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.131614 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:19.131861 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:19.132182 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 19:29:19.132405 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260923 19:29:19.133538 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46084
I20260923 19:29:19.136089 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1585
I20260923 19:29:19.158071 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:19.137004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.0.97.1:33059)
0923 19:29:19.137224 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:19.137231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:19.144270 (+ 7039us) server_negotiation.cc:407] Connection header received
0923 19:29:19.144410 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:19.144415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:19.144506 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:19.144626 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:19.145918 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.147074 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.148548 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.149182 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.149313 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:19.152010 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:19.152016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:19.152022 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:19.152060 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:19.155437 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.156108 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.156114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.156117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.156269 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.156662 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.156666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.156670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.156968 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:19.157226 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:19.157499 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 19:29:19.157716 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260923 19:29:19.158789 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49736
W20260923 19:29:19.159463 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:29:19.159680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 1585
I20260923 19:29:19.175971 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1725
Sep 23 19:29:19 dist-test-slave-kxfb krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:29:19.193216 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:19.176924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.0.97.2:36075)
0923 19:29:19.177130 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:19.177136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:19.177204 (+ 68us) server_negotiation.cc:407] Connection header received
0923 19:29:19.177476 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:19.177481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:19.177555 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:19.177657 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:19.178941 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.179811 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.181051 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.181553 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.181661 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:19.186488 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:19.186503 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:19.186522 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:19.186557 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:19.190173 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.190988 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.190998 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.191002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.191165 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.191535 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.191538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.191540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.191959 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:19.192291 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:19.192566 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 19:29:19.192824 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 19:29:19.194034 1855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42026
W20260923 19:29:19.195048 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:29:19.195248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 1725
I20260923 19:29:19.212793 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1864
Sep 23 19:29:19 dist-test-slave-kxfb krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:29:19.229645 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:19.213693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.0.97.3:45889)
0923 19:29:19.213921 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:19.213926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:19.213947 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:29:19.214077 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:19.214081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:19.214149 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:19.214242 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:19.215762 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.216485 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.217906 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:19.218568 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:19.218691 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:19.223428 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:19.223448 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:19.223469 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:19.223522 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:19.226639 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.227472 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.227480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.227483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.227664 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:19.228139 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:19.228143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:19.228146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:19.228589 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:19.228878 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:19.229131 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 19:29:19.229360 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260923 19:29:19.230619 1994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34682
W20260923 19:29:19.231300 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:29:19.231524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 1864
I20260923 19:29:19.249094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 1482
I20260923 19:29:19.250665 1554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:19.364544 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 1482
2026-09-23T19:29:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4965 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:19 dist-test-slave-kxfb krb5kdc[2093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:19 dist-test-slave-kxfb krb5kdc[2093](info): set up 2 sockets
Sep 23 19:29:19 dist-test-slave-kxfb krb5kdc[2093](info): commencing operation
krb5kdc: starting...
W20260923 19:29:20.362769 388 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:20 dist-test-slave-kxfb krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:20.563004 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38485
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:20.894182 2106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:20.894531 2106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:20.894670 2106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:20.901873 2106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:20.901993 2106 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 19:29:20.902057 2106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:20.902145 2106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:20.902209 2106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:20.902311 2106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:20.912214 2106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38485
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:20.913472 2106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:20.914772 2106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:20.926764 2111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:20.927148 2112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:20.927213 2114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:20.928208 2106 server_base.cc:1034] running on GCE node
I20260923 19:29:20.928972 2106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:20.930426 2106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:20.931658 2106 hybrid_clock.cc:648] HybridClock initialized: now 1790191760931579 us; error 66 us; skew 500 ppm
I20260923 19:29:20.931978 2106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:20 dist-test-slave-kxfb krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:20.935055 2106 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:20.936606 2106 webserver.cc:460] Webserver started at http://127.0.97.62:42115/ using document root <none> and password file <none>
I20260923 19:29:20.937294 2106 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:20.937387 2106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:20.937669 2106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:20.940589 2106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "843f2e85983e45408654a1c3727eeaa1"
format_stamp: "Formatted at 2026-09-23 19:29:20 on dist-test-slave-kxfb"
server_key: "525c6dd105aaae13648e7951a29fc397"
server_key_iv: "7f62829a72d079999f97003ebebf5e12"
server_key_version: "encryptionkey@0"
I20260923 19:29:20.941332 2106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "843f2e85983e45408654a1c3727eeaa1"
format_stamp: "Formatted at 2026-09-23 19:29:20 on dist-test-slave-kxfb"
server_key: "525c6dd105aaae13648e7951a29fc397"
server_key_iv: "7f62829a72d079999f97003ebebf5e12"
server_key_version: "encryptionkey@0"
I20260923 19:29:20.946617 2106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 19:29:20.950328 2121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:20.951894 2106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:20.952052 2106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "843f2e85983e45408654a1c3727eeaa1"
format_stamp: "Formatted at 2026-09-23 19:29:20 on dist-test-slave-kxfb"
server_key: "525c6dd105aaae13648e7951a29fc397"
server_key_iv: "7f62829a72d079999f97003ebebf5e12"
server_key_version: "encryptionkey@0"
I20260923 19:29:20.952275 2106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:20.972551 2106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:20.976666 2106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:20.976918 2106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:20.999226 2106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38485
I20260923 19:29:20.999271 2182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38485 every 8 connection(s)
I20260923 19:29:21.000547 2106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:21.002207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 2106
I20260923 19:29:21.002447 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:21.003062 388 external_mini_cluster.cc:1428] Setting key 787647fb2f8084394ea4537b88b5e9bd
I20260923 19:29:21.005970 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 19:29:21 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:21.021456 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1: Bootstrap starting.
I20260923 19:29:21.026908 2183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:21.028318 2183 log.cc:824] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:21.029896 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:21.005161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.0.97.62:38485)
0923 19:29:21.005970 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:21.005982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:21.006024 (+ 42us) server_negotiation.cc:407] Connection header received
0923 19:29:21.007738 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:21.007760 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:21.008187 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:21.008520 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:21.010733 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.012323 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.014420 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.015247 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.015423 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:21.020792 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:21.020801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:21.020820 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:21.020866 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:21.024980 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:21.025937 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:21.025944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:21.025952 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:21.026132 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:21.026554 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:21.026560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:21.026564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:21.027236 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:21.027559 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:21.028001 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 19:29:21.028462 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":354,"threads_started":1}
I20260923 19:29:21.032294 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1: No bootstrap required, opened a new log
I20260923 19:29:21.040792 2183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f2e85983e45408654a1c3727eeaa1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } }
I20260923 19:29:21.041337 2183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:21.041496 2183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 843f2e85983e45408654a1c3727eeaa1, State: Initialized, Role: FOLLOWER
I20260923 19:29:21.042330 2183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f2e85983e45408654a1c3727eeaa1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } }
I20260923 19:29:21.042625 2183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:21.042773 2183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:21.042966 2183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:21.046551 2183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f2e85983e45408654a1c3727eeaa1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } }
I20260923 19:29:21.047320 2183 leader_election.cc:302] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 843f2e85983e45408654a1c3727eeaa1; no voters:
I20260923 19:29:21.048099 2183 leader_election.cc:288] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:21.048285 2188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:21.050642 2188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [term 1 LEADER]: Becoming Leader. State: Replica: 843f2e85983e45408654a1c3727eeaa1, State: Running, Role: LEADER
I20260923 19:29:21.050776 2183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:21.051393 2188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f2e85983e45408654a1c3727eeaa1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } }
I20260923 19:29:21.054719 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 843f2e85983e45408654a1c3727eeaa1. Latest consensus state: current_term: 1 leader_uuid: "843f2e85983e45408654a1c3727eeaa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f2e85983e45408654a1c3727eeaa1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } } }
I20260923 19:29:21.055224 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:21.054744 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "843f2e85983e45408654a1c3727eeaa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f2e85983e45408654a1c3727eeaa1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } } }
I20260923 19:29:21.055505 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:21.057653 2197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:21.062212 2197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:21.071044 2197 catalog_manager.cc:1347] Generated new cluster ID: 00d752bc27ac47a8b95489d4a26ca0de
I20260923 19:29:21.071204 2197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:21.086241 2197 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:21.087819 2197 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:21.097772 2197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1: Generated new TSK 0
I20260923 19:29:21.098673 2197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:29:21.150606 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38485
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:21.489854 2209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:21.490155 2209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:21.490279 2209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:21.497435 2209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:21.497555 2209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:21.497665 2209 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 19:29:21.497732 2209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 19:29:21.497833 2209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:21.507664 2209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:21.508913 2209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:21.510176 2209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:21.523018 2214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:21.523144 2215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:21.523164 2217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:21.524403 2209 server_base.cc:1034] running on GCE node
I20260923 19:29:21.525141 2209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:21.526226 2209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:21.527484 2209 hybrid_clock.cc:648] HybridClock initialized: now 1790191761527406 us; error 64 us; skew 500 ppm
I20260923 19:29:21.527837 2209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:21 dist-test-slave-kxfb krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:21.531395 2209 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:29:21.533087 2209 webserver.cc:460] Webserver started at http://127.0.97.1:42941/ using document root <none> and password file <none>
I20260923 19:29:21.533917 2209 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:21.534044 2209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:21.534416 2209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:21.537710 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "5119edcb73284b7297435397958a1abd"
format_stamp: "Formatted at 2026-09-23 19:29:21 on dist-test-slave-kxfb"
server_key: "cb8807bdfa34dfbf8dd46cd668590257"
server_key_iv: "b7e376db4c4be5472139d6b6f884e756"
server_key_version: "encryptionkey@0"
I20260923 19:29:21.538604 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5119edcb73284b7297435397958a1abd"
format_stamp: "Formatted at 2026-09-23 19:29:21 on dist-test-slave-kxfb"
server_key: "cb8807bdfa34dfbf8dd46cd668590257"
server_key_iv: "b7e376db4c4be5472139d6b6f884e756"
server_key_version: "encryptionkey@0"
I20260923 19:29:21.544534 2209 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 19:29:21.548547 2224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:21.550171 2209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 19:29:21.550376 2209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "5119edcb73284b7297435397958a1abd"
format_stamp: "Formatted at 2026-09-23 19:29:21 on dist-test-slave-kxfb"
server_key: "cb8807bdfa34dfbf8dd46cd668590257"
server_key_iv: "b7e376db4c4be5472139d6b6f884e756"
server_key_version: "encryptionkey@0"
I20260923 19:29:21.550595 2209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:21.584296 2209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:21.589114 2209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:21.589425 2209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:21.592607 2209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:21.592717 2209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:21.592844 2209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:21.592919 2209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 19:29:21 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:21.618427 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:21.593897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.97.1:60843)
0923 19:29:21.594847 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:21.594904 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:21.597032 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:21.598204 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:21.598211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:21.599020 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:21.599885 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:21.599907 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.601155 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.601160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:21.603338 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:21.603351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.604357 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.604368 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:21.604618 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:21.605577 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:21.605606 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:21.608799 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:21.613122 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:21.613130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:21.613143 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:21.613589 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:21.614107 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:21.614112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:21.614116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:21.614453 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:21.615189 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:21.615198 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:21.615673 (+ 475us) client_negotiation.cc:769] Sending connection context
0923 19:29:21.616007 (+ 334us) client_negotiation.cc:240] Negotiation successful
0923 19:29:21.616634 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":422,"threads_started":1}
I20260923 19:29:21.619472 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:21.593303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60843 (local address 127.0.97.62:38485)
0923 19:29:21.593834 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:21.593841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:21.595052 (+ 1211us) server_negotiation.cc:407] Connection header received
0923 19:29:21.597575 (+ 2523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:21.597582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:21.597663 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:21.597741 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:21.600112 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.600988 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.603501 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.603919 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.604080 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:21.609183 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:21.609194 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:21.609201 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:21.609257 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:21.612945 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:21.613747 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:21.613754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:21.613759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:21.613930 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:21.614615 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:21.614620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:21.614623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:21.615024 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:21.618484 (+ 3460us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:21.618696 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 19:29:21.619007 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":271,"threads_started":1}
I20260923 19:29:21.636492 2209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41265
I20260923 19:29:21.636535 2344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41265 every 8 connection(s)
I20260923 19:29:21.638226 2209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:21.639164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 2209
I20260923 19:29:21.639640 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:21.640393 388 external_mini_cluster.cc:1428] Setting key e1a22d97d01ef595a7fe46fc4273287d
I20260923 19:29:21.640885 2345 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38485
I20260923 19:29:21.641206 2345 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:21.642369 2345 heartbeater.cc:511] Master 127.0.97.62:38485 requested a full tablet report, sending...
I20260923 19:29:21.644953 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:21.627757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52121 (local address 127.0.97.62:38485)
0923 19:29:21.627990 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:21.627997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:21.628666 (+ 669us) server_negotiation.cc:407] Connection header received
0923 19:29:21.628735 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:21.628740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:21.628824 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:21.628926 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:21.630819 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.631903 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.634280 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.634790 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.634930 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:21.637653 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:21.637660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:21.637666 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:21.637701 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:21.641237 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:21.642318 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:21.642324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:21.642328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:21.642512 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:21.642962 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:21.642968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:21.642970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:21.643381 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:21.643553 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:21.644218 (+ 665us) server_negotiation.cc:299] Negotiation successful
0923 19:29:21.644525 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 19:29:21.646246 2323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:21.627694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.97.1:52121)
0923 19:29:21.628348 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:21.628367 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:21.628542 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:21.629023 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:21.629029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:21.629908 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:21.630612 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:21.630630 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.632078 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.632084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:21.634086 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:21.634101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:21.635035 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:21.635044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:21.635241 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:21.636183 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:21.636210 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:21.637428 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:21.641475 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:21.641481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:21.641487 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:21.642138 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:21.642671 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:21.642676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:21.642681 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:21.642850 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:21.643655 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:21.643661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:21.643891 (+ 230us) client_negotiation.cc:769] Sending connection context
0923 19:29:21.645279 (+ 1388us) client_negotiation.cc:240] Negotiation successful
0923 19:29:21.645725 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":351,"threads_started":1}
I20260923 19:29:21.647900 2138 ts_manager.cc:194] Registered new tserver with Master: 5119edcb73284b7297435397958a1abd (127.0.97.1:41265)
I20260923 19:29:21.649434 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60843
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:29:21.686769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38485
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:22.045841 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:22.046128 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:22.046227 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:22.053408 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:22.053522 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:22.053604 2349 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 19:29:22.053658 2349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 19:29:22.053726 2349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:22.063692 2349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:22.065047 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:22.066388 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:22.079761 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:22.079782 2357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:22.080166 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:22.081344 2349 server_base.cc:1034] running on GCE node
I20260923 19:29:22.081933 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:22.082984 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:22.084199 2349 hybrid_clock.cc:648] HybridClock initialized: now 1790191762084140 us; error 59 us; skew 500 ppm
I20260923 19:29:22.084513 2349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:22 dist-test-slave-kxfb krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:22.087684 2349 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:29:22.089172 2349 webserver.cc:460] Webserver started at http://127.0.97.2:38315/ using document root <none> and password file <none>
I20260923 19:29:22.089857 2349 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:22.089953 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:22.090207 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:22.093086 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "68aeb6578f474046943d0466adbeed9e"
format_stamp: "Formatted at 2026-09-23 19:29:22 on dist-test-slave-kxfb"
server_key: "7fdd44b51bc7f7daa64932c5ca0fe9c0"
server_key_iv: "096a70695dea4525d2d8aacde6dfb656"
server_key_version: "encryptionkey@0"
I20260923 19:29:22.093834 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "68aeb6578f474046943d0466adbeed9e"
format_stamp: "Formatted at 2026-09-23 19:29:22 on dist-test-slave-kxfb"
server_key: "7fdd44b51bc7f7daa64932c5ca0fe9c0"
server_key_iv: "096a70695dea4525d2d8aacde6dfb656"
server_key_version: "encryptionkey@0"
I20260923 19:29:22.099248 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 19:29:22.103016 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:22.104619 2349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 19:29:22.104771 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "68aeb6578f474046943d0466adbeed9e"
format_stamp: "Formatted at 2026-09-23 19:29:22 on dist-test-slave-kxfb"
server_key: "7fdd44b51bc7f7daa64932c5ca0fe9c0"
server_key_iv: "096a70695dea4525d2d8aacde6dfb656"
server_key_version: "encryptionkey@0"
I20260923 19:29:22.104933 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:22.123045 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:22.127530 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:22.127810 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:22.130659 2349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:22.130762 2349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:22.130870 2349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:22.130940 2349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 19:29:22 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:22.157038 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.131723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33751 (local address 127.0.97.62:38485)
0923 19:29:22.131984 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.131991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.133621 (+ 1630us) server_negotiation.cc:407] Connection header received
0923 19:29:22.136522 (+ 2901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.136530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.136629 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.136760 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:22.139284 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.140352 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.143566 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.144258 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.144403 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.149917 (+ 5514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:22.149926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:22.149931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:22.149982 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:22.153081 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.153947 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.153951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.153954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.154072 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.154851 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.154855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.154858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.155222 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:22.155386 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.156391 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.156650 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 19:29:22.158377 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.132351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.97.2:33751)
0923 19:29:22.133396 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:22.133460 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:22.136081 (+ 2621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:22.137136 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:22.137152 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:22.137991 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:22.139059 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.139086 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.140626 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.140637 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.143301 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.143327 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.144566 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.144575 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.144920 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.145842 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:22.145867 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:22.149584 (+ 3717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:22.153247 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:22.153258 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:22.153278 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:22.153809 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:22.154231 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:22.154237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:22.154245 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:22.154688 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:22.155481 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:22.155488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:22.155990 (+ 502us) client_negotiation.cc:769] Sending connection context
0923 19:29:22.156275 (+ 285us) client_negotiation.cc:240] Negotiation successful
0923 19:29:22.156921 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":467,"threads_started":1}
I20260923 19:29:22.184751 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.165132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45017 (local address 127.0.97.62:38485)
0923 19:29:22.165368 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.165376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.166015 (+ 639us) server_negotiation.cc:407] Connection header received
0923 19:29:22.166244 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.166249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.166372 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.166478 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:22.168001 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.169040 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.171223 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.171674 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.171796 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.177813 (+ 6017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:22.177820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:22.177824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:22.177868 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:22.181493 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.182589 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.182595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.182599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.182731 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.183129 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.183132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.183134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.183494 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:22.183615 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.184301 (+ 686us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.184494 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 19:29:22.185133 2436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.165078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.97.2:45017)
0923 19:29:22.165872 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:22.165897 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:22.166097 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:22.166544 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:22.166549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:22.167391 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:22.167860 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.167872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.169185 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.169192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.171083 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.171099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.171873 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.171879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.172043 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.176102 (+ 4059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:22.176145 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:22.177519 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:22.181736 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:22.181742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:22.181747 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:22.182434 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:22.182847 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:22.182852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:22.182856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:22.183017 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:22.183676 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:22.183683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:22.183966 (+ 283us) client_negotiation.cc:769] Sending connection context
0923 19:29:22.184187 (+ 221us) client_negotiation.cc:240] Negotiation successful
0923 19:29:22.184590 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":398,"threads_started":1}
I20260923 19:29:22.188735 2349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38407
I20260923 19:29:22.188839 2483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38407 every 8 connection(s)
I20260923 19:29:22.189991 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:22.192519 2484 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38485
I20260923 19:29:22.192862 2484 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:22.193627 2484 heartbeater.cc:511] Master 127.0.97.62:38485 requested a full tablet report, sending...
I20260923 19:29:22.195180 2138 ts_manager.cc:194] Registered new tserver with Master: 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407)
I20260923 19:29:22.196087 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33751
I20260923 19:29:22.197206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 2349
I20260923 19:29:22.197381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:22.197852 388 external_mini_cluster.cc:1428] Setting key 55f76e9f31edddf08c6318efe025c3ea
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:29:22.243697 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38485
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:22.567468 2488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:22.567750 2488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:22.567876 2488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:22.574970 2488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:22.575076 2488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:22.575181 2488 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 19:29:22.575256 2488 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 19:29:22.575363 2488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:22.584851 2488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:22.586107 2488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:22.587394 2488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:22.599895 2493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:22.600147 2494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:22.600634 2496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:22.601402 2488 server_base.cc:1034] running on GCE node
I20260923 19:29:22.602212 2488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:22.603330 2488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:22.604562 2488 hybrid_clock.cc:648] HybridClock initialized: now 1790191762604513 us; error 34 us; skew 500 ppm
I20260923 19:29:22.604926 2488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:22 dist-test-slave-kxfb krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:22.608485 2488 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:29:22.610164 2488 webserver.cc:460] Webserver started at http://127.0.97.3:35329/ using document root <none> and password file <none>
I20260923 19:29:22.610996 2488 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:22.611116 2488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:22.611438 2488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:22.614778 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "d3243e03e96e4dde814f8c392e9d0684"
format_stamp: "Formatted at 2026-09-23 19:29:22 on dist-test-slave-kxfb"
server_key: "fdc9df02c917292ea16e936ac0f103b0"
server_key_iv: "647196045446d777acc6d157210d9db1"
server_key_version: "encryptionkey@0"
I20260923 19:29:22.615621 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d3243e03e96e4dde814f8c392e9d0684"
format_stamp: "Formatted at 2026-09-23 19:29:22 on dist-test-slave-kxfb"
server_key: "fdc9df02c917292ea16e936ac0f103b0"
server_key_iv: "647196045446d777acc6d157210d9db1"
server_key_version: "encryptionkey@0"
I20260923 19:29:22.621526 2488 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 19:29:22.625614 2503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:22.627296 2488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:22.627470 2488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "d3243e03e96e4dde814f8c392e9d0684"
format_stamp: "Formatted at 2026-09-23 19:29:22 on dist-test-slave-kxfb"
server_key: "fdc9df02c917292ea16e936ac0f103b0"
server_key_iv: "647196045446d777acc6d157210d9db1"
server_key_version: "encryptionkey@0"
I20260923 19:29:22.627674 2488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:22.641898 2488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:22.646663 2488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:22.646968 2488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:22.649924 2488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:22.650033 2488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:22.650154 2488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:22.650243 2488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:22.652937 2345 heartbeater.cc:503] Master 127.0.97.62:38485 was elected leader, sending a full tablet report...
Sep 23 19:29:22 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:22.676117 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.650719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37201 (local address 127.0.97.62:38485)
0923 19:29:22.650920 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.650926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.652614 (+ 1688us) server_negotiation.cc:407] Connection header received
0923 19:29:22.656014 (+ 3400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.656019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.656109 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.656214 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:22.659299 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.660209 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.662795 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.663576 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.663747 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.669171 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:22.669177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:22.669181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:22.669221 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:22.672331 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.673381 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.673386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.673390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.673508 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.674054 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.674059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.674063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.674457 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:22.674615 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.675530 (+ 915us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.675766 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 19:29:22.677186 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.651350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.97.3:37201)
0923 19:29:22.652396 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:22.652462 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:22.655677 (+ 3215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:22.656679 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:22.656698 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:22.657774 (+ 1076us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:22.659061 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.659096 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.660436 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.660442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.662585 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.662604 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.663869 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.663877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.664185 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.665135 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:22.665166 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:22.668823 (+ 3657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:22.672504 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:22.672519 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:22.672543 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:22.673204 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:22.673637 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:22.673643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:22.673647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:22.673927 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:22.674573 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:22.674581 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:22.675118 (+ 537us) client_negotiation.cc:769] Sending connection context
0923 19:29:22.675374 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 19:29:22.675937 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":371,"threads_started":1}
I20260923 19:29:22.699553 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.683099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50563 (local address 127.0.97.62:38485)
0923 19:29:22.683340 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.683348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.684134 (+ 786us) server_negotiation.cc:407] Connection header received
0923 19:29:22.684278 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.684285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.684403 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.684572 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:22.685398 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.686646 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.688476 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.689087 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.689226 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.692620 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:22.692625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:22.692628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:22.692660 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:22.696203 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.697335 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.697341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.697345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.697512 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.698028 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.698034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.698037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.698391 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:22.698546 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.698974 (+ 428us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.699211 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 19:29:22.699810 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.683101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.97.3:50563)
0923 19:29:22.684004 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:22.684038 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:22.684155 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:22.684589 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:22.684592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:22.684977 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:22.685249 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.685258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.686823 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.686828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.688320 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.688332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.689361 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.689374 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.689685 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.690939 (+ 1254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:22.690970 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:22.692396 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:22.696418 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:22.696427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:22.696432 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:22.697135 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:22.697676 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:22.697683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:22.697691 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:22.697891 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:22.698509 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:22.698515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:22.698716 (+ 201us) client_negotiation.cc:769] Sending connection context
0923 19:29:22.698844 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 19:29:22.699261 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":475,"threads_started":1}
I20260923 19:29:22.710812 2488 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33515
I20260923 19:29:22.710853 2622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33515 every 8 connection(s)
I20260923 19:29:22.712267 2488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:22.713578 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 2488
I20260923 19:29:22.713778 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:22.714331 388 external_mini_cluster.cc:1428] Setting key d7e3f528e33d03048b44b940eadb299a
I20260923 19:29:22.714955 2623 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38485
I20260923 19:29:22.715181 2623 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:22.715942 2623 heartbeater.cc:511] Master 127.0.97.62:38485 requested a full tablet report, sending...
I20260923 19:29:22.717387 2137 ts_manager.cc:194] Registered new tserver with Master: d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515)
I20260923 19:29:22.718268 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37201
I20260923 19:29:22.730463 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 19:29:22 dist-test-slave-kxfb krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 19:29:22 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:22.781217 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.766705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.0.97.62:38485)
0923 19:29:22.766909 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.766914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.767385 (+ 471us) server_negotiation.cc:407] Connection header received
0923 19:29:22.767529 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.767534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.767610 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.767722 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:22.769342 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.770080 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.771540 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.771928 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.772029 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.776072 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:22.776077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:22.776080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:22.776114 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:22.778698 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.779384 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.779389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.779393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.779540 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:22.779907 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:22.779913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:22.779916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:22.780254 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:22.780458 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.780757 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.780924 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 19:29:22.785626 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:29:22.792282 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:22.829581 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.806018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.0.97.1:41265)
0923 19:29:22.818255 (+ 12237us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.818265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.819583 (+ 1318us) server_negotiation.cc:407] Connection header received
0923 19:29:22.819719 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.819729 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.820031 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.820258 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:22.821418 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.823273 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.826664 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.828036 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.828163 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.828366 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.829077 (+ 711us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.829259 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12063,"thread_start_us":11901,"threads_started":1}
I20260923 19:29:22.829608 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.806068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.0.97.3:33515)
0923 19:29:22.817461 (+ 11393us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.817474 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.818979 (+ 1505us) server_negotiation.cc:407] Connection header received
0923 19:29:22.819296 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.819305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.819571 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.819756 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:22.820964 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.822666 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.826459 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.827978 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.828105 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.828390 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.829062 (+ 672us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.829244 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11098,"thread_start_us":11421,"threads_started":1}
I20260923 19:29:22.830060 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.805708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33515 (local address 127.0.0.1:38822)
0923 19:29:22.818793 (+ 13085us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:22.818838 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:22.819023 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:22.820152 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:22.820161 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:22.820197 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:22.820682 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.820701 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.822869 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.822874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.825707 (+ 2833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.825729 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.828552 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.828558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.828784 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.828868 (+ 84us) client_negotiation.cc:769] Sending connection context
0923 19:29:22.828997 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 19:29:22.829215 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12805,"mutex_wait_us":36,"thread_start_us":12642,"threads_started":1}
I20260923 19:29:22.830530 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.805900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41265 (local address 127.0.0.1:40794)
0923 19:29:22.819263 (+ 13363us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:22.819292 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:22.819444 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:22.820521 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:22.820527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:22.820556 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:22.821159 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.821171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.823491 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.823496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.826494 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.826507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.828506 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.828528 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.828786 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.828866 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 19:29:22.829859 (+ 993us) client_negotiation.cc:240] Negotiation successful
0923 19:29:22.830016 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13235,"mutex_wait_us":411,"thread_start_us":12397,"threads_started":1}
I20260923 19:29:22.833801 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.819346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38407 (local address 127.0.0.1:34592)
0923 19:29:22.825971 (+ 6625us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:22.825995 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:22.826127 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:22.827098 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:22.827103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:22.827133 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:22.827491 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.827502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.829235 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.829240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.831795 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.831807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.833115 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.833123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.833284 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.833307 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 19:29:22.833385 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 19:29:22.833507 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6491,"thread_start_us":6325,"threads_started":1}
I20260923 19:29:22.833885 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.819567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.0.97.2:38407)
0923 19:29:22.826376 (+ 6809us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.826385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.826415 (+ 30us) server_negotiation.cc:407] Connection header received
0923 19:29:22.826529 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.826538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.826834 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.827085 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:22.827715 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.829078 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.831947 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.832857 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.832973 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.833155 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.833456 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.833631 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6610,"thread_start_us":6460,"threads_started":1}
I20260923 19:29:22.834771 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 4e772f14204a4e3d888d173fe3917777 (DEFAULT_TABLE table=test-table [id=4f8c8e5f2a5e42a7b0c38877ed5fa5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:22.836565 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e772f14204a4e3d888d173fe3917777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:22.837373 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 4e772f14204a4e3d888d173fe3917777 (DEFAULT_TABLE table=test-table [id=4f8c8e5f2a5e42a7b0c38877ed5fa5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:22.838353 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 4e772f14204a4e3d888d173fe3917777 (DEFAULT_TABLE table=test-table [id=4f8c8e5f2a5e42a7b0c38877ed5fa5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:22.840108 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e772f14204a4e3d888d173fe3917777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:22.840231 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e772f14204a4e3d888d173fe3917777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:22.849246 2642 tablet_bootstrap.cc:492] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: Bootstrap starting.
I20260923 19:29:22.852937 2642 tablet_bootstrap.cc:654] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:22.854123 2642 log.cc:824] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:22.856263 2643 tablet_bootstrap.cc:492] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: Bootstrap starting.
I20260923 19:29:22.856446 2644 tablet_bootstrap.cc:492] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: Bootstrap starting.
I20260923 19:29:22.859941 2644 tablet_bootstrap.cc:654] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:22.859938 2643 tablet_bootstrap.cc:654] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:22.860229 2642 tablet_bootstrap.cc:492] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: No bootstrap required, opened a new log
I20260923 19:29:22.860555 2642 ts_tablet_manager.cc:1397] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260923 19:29:22.861081 2644 log.cc:824] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:22.861250 2643 log.cc:824] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:22.863979 2644 tablet_bootstrap.cc:492] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: No bootstrap required, opened a new log
I20260923 19:29:22.864269 2644 ts_tablet_manager.cc:1397] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 19:29:22.868897 2643 tablet_bootstrap.cc:492] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: No bootstrap required, opened a new log
I20260923 19:29:22.869225 2643 ts_tablet_manager.cc:1397] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260923 19:29:22.869314 2642 raft_consensus.cc:348] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } }
I20260923 19:29:22.869801 2642 raft_consensus.cc:374] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:22.869904 2642 raft_consensus.cc:729] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5119edcb73284b7297435397958a1abd, State: Initialized, Role: FOLLOWER
I20260923 19:29:22.870675 2642 consensus_queue.cc:260] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } }
I20260923 19:29:22.871759 2644 raft_consensus.cc:348] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } }
I20260923 19:29:22.872195 2644 raft_consensus.cc:374] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:22.872287 2644 raft_consensus.cc:729] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3243e03e96e4dde814f8c392e9d0684, State: Initialized, Role: FOLLOWER
I20260923 19:29:22.872978 2644 consensus_queue.cc:260] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } }
I20260923 19:29:22.873937 2623 heartbeater.cc:503] Master 127.0.97.62:38485 was elected leader, sending a full tablet report...
I20260923 19:29:22.875257 2644 ts_tablet_manager.cc:1428] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260923 19:29:22.877828 2642 ts_tablet_manager.cc:1428] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: Time spent starting tablet: real 0.017s user 0.008s sys 0.004s
I20260923 19:29:22.879155 2643 raft_consensus.cc:348] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } }
I20260923 19:29:22.879561 2643 raft_consensus.cc:374] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:22.879683 2643 raft_consensus.cc:729] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aeb6578f474046943d0466adbeed9e, State: Initialized, Role: FOLLOWER
I20260923 19:29:22.880237 2643 consensus_queue.cc:260] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } }
I20260923 19:29:22.881291 2484 heartbeater.cc:503] Master 127.0.97.62:38485 was elected leader, sending a full tablet report...
I20260923 19:29:22.881841 2643 ts_tablet_manager.cc:1428] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: Time spent starting tablet: real 0.012s user 0.004s sys 0.010s
W20260923 19:29:22.904313 2346 tablet.cc:2369] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:29:22.936008 2650 raft_consensus.cc:482] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:22.936399 2650 raft_consensus.cc:504] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } }
I20260923 19:29:22.938119 2650 leader_election.cc:288] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515), 5119edcb73284b7297435397958a1abd (127.0.97.1:41265)
W20260923 19:29:22.942195 2485 tablet.cc:2369] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:29:22.945326 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.938628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35273 (local address 127.0.97.3:33515)
0923 19:29:22.938822 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.938827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.939577 (+ 750us) server_negotiation.cc:407] Connection header received
0923 19:29:22.939809 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.939813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.939882 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.939961 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:22.940536 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.941282 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.943458 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.944374 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.944502 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.944620 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.944899 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.945034 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 19:29:22.945464 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.938533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33515 (local address 127.0.97.2:35273)
0923 19:29:22.939455 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:22.939471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:22.939649 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:22.940001 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:22.940005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:22.940030 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:22.940399 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.940409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.941417 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.941423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.943303 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.943316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.944602 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.944609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.944749 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.944774 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 19:29:22.944874 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 19:29:22.945021 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":6,"thread_start_us":967,"threads_started":1}
I20260923 19:29:22.945531 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.938696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51497 (local address 127.0.97.1:41265)
0923 19:29:22.938927 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:22.938934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:22.939969 (+ 1035us) server_negotiation.cc:407] Connection header received
0923 19:29:22.940148 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:22.940153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:22.940230 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:22.940305 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:22.940854 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.941564 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.943842 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.944699 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.944827 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.944926 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:22.945196 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 19:29:22.945313 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 19:29:22.945870 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:22.938585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41265 (local address 127.0.97.2:51497)
0923 19:29:22.939836 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:22.939853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:22.940006 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:22.940353 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:22.940357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:22.940385 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:22.940716 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.940725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.941702 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.941709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.943695 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:22.943707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:22.944917 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:22.944922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:22.945050 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:22.945075 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 19:29:22.945158 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 19:29:22.945296 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"thread_start_us":455,"threads_started":1}
I20260923 19:29:22.946743 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e772f14204a4e3d888d173fe3917777" candidate_uuid: "68aeb6578f474046943d0466adbeed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684" is_pre_election: true
I20260923 19:29:22.946939 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e772f14204a4e3d888d173fe3917777" candidate_uuid: "68aeb6578f474046943d0466adbeed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5119edcb73284b7297435397958a1abd" is_pre_election: true
I20260923 19:29:22.947218 2578 raft_consensus.cc:2454] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aeb6578f474046943d0466adbeed9e in term 0.
I20260923 19:29:22.947531 2295 raft_consensus.cc:2454] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aeb6578f474046943d0466adbeed9e in term 0.
I20260923 19:29:22.947957 2367 leader_election.cc:302] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aeb6578f474046943d0466adbeed9e, d3243e03e96e4dde814f8c392e9d0684; no voters:
I20260923 19:29:22.948403 2650 raft_consensus.cc:2781] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:22.948558 2650 raft_consensus.cc:482] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:22.948647 2650 raft_consensus.cc:3037] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:22.951061 2650 raft_consensus.cc:504] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } }
I20260923 19:29:22.951953 2650 leader_election.cc:288] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [CANDIDATE]: Term 1 election: Requested vote from peers d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515), 5119edcb73284b7297435397958a1abd (127.0.97.1:41265)
I20260923 19:29:22.952525 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e772f14204a4e3d888d173fe3917777" candidate_uuid: "68aeb6578f474046943d0466adbeed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684"
I20260923 19:29:22.952513 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e772f14204a4e3d888d173fe3917777" candidate_uuid: "68aeb6578f474046943d0466adbeed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5119edcb73284b7297435397958a1abd"
I20260923 19:29:22.952721 2578 raft_consensus.cc:3037] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:22.952742 2295 raft_consensus.cc:3037] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:22.956000 2578 raft_consensus.cc:2454] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aeb6578f474046943d0466adbeed9e in term 1.
I20260923 19:29:22.956005 2295 raft_consensus.cc:2454] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aeb6578f474046943d0466adbeed9e in term 1.
I20260923 19:29:22.956485 2367 leader_election.cc:302] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5119edcb73284b7297435397958a1abd, 68aeb6578f474046943d0466adbeed9e, d3243e03e96e4dde814f8c392e9d0684; no voters:
I20260923 19:29:22.956820 2650 raft_consensus.cc:2781] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:22.957367 2650 raft_consensus.cc:686] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 1 LEADER]: Becoming Leader. State: Replica: 68aeb6578f474046943d0466adbeed9e, State: Running, Role: LEADER
I20260923 19:29:22.957934 2650 consensus_queue.cc:237] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } }
I20260923 19:29:22.963445 2138 catalog_manager.cc:5495] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 68aeb6578f474046943d0466adbeed9e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "68aeb6578f474046943d0466adbeed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } health_report { overall_health: HEALTHY } } }
W20260923 19:29:22.963971 2624 tablet.cc:2369] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 19:29:23 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:23.047617 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.034200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.97.62:38485)
0923 19:29:23.034454 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:23.034461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:23.034774 (+ 313us) server_negotiation.cc:407] Connection header received
0923 19:29:23.034971 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:23.034977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:23.035043 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:23.035124 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:23.036406 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.037171 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.038457 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.038891 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.038995 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.042439 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:23.042444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:23.042447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:23.042474 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:23.044959 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:23.045638 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:23.045642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:23.045644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:23.045766 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:23.046112 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:23.046115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:23.046117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:23.046460 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:23.046592 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:23.047004 (+ 412us) server_negotiation.cc:299] Negotiation successful
0923 19:29:23.047297 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":108}
I20260923 19:29:23.047660 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.033903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.0.1:43362)
0923 19:29:23.034651 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:23.034668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:23.034846 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:23.035165 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:23.035169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:23.035975 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:23.036252 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.036261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.037304 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.037309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.038328 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.038342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.039065 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.039071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.039222 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.039878 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:23.039895 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:23.042251 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:23.045075 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:23.045087 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:23.045096 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:23.045522 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:23.045874 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:23.045879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:23.045883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:23.046009 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:23.046584 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:23.046594 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:23.046749 (+ 155us) client_negotiation.cc:769] Sending connection context
0923 19:29:23.046884 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 19:29:23.047164 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":456,"threads_started":1}
I20260923 19:29:23.053562 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43362:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 19:29:23.054764 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:23.062554 2414 tablet_service.cc:1467] Processing CreateTablet for tablet e0d34a3469ce4de4b4732f95f5f9e61a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c00a0db463524744afa8a9c620e4dd5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 19:29:23.062567 2275 tablet_service.cc:1467] Processing CreateTablet for tablet e0d34a3469ce4de4b4732f95f5f9e61a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c00a0db463524744afa8a9c620e4dd5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 19:29:23.062721 2553 tablet_service.cc:1467] Processing CreateTablet for tablet e0d34a3469ce4de4b4732f95f5f9e61a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c00a0db463524744afa8a9c620e4dd5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 19:29:23.063565 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d34a3469ce4de4b4732f95f5f9e61a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:23.063572 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d34a3469ce4de4b4732f95f5f9e61a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:23.063843 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d34a3469ce4de4b4732f95f5f9e61a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:23.069456 2644 tablet_bootstrap.cc:492] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: Bootstrap starting.
I20260923 19:29:23.070686 2643 tablet_bootstrap.cc:492] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: Bootstrap starting.
I20260923 19:29:23.070812 2642 tablet_bootstrap.cc:492] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: Bootstrap starting.
I20260923 19:29:23.071139 2644 tablet_bootstrap.cc:654] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:23.072397 2643 tablet_bootstrap.cc:654] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:23.072546 2642 tablet_bootstrap.cc:654] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:23.073824 2644 tablet_bootstrap.cc:492] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: No bootstrap required, opened a new log
I20260923 19:29:23.074002 2644 ts_tablet_manager.cc:1397] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 19:29:23.074924 2644 raft_consensus.cc:348] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.075212 2644 raft_consensus.cc:374] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:23.075307 2644 raft_consensus.cc:729] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3243e03e96e4dde814f8c392e9d0684, State: Initialized, Role: FOLLOWER
I20260923 19:29:23.075600 2644 consensus_queue.cc:260] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.076134 2649 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.076601 2649 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:23.076781 2642 tablet_bootstrap.cc:492] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: No bootstrap required, opened a new log
I20260923 19:29:23.076835 2643 tablet_bootstrap.cc:492] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: No bootstrap required, opened a new log
I20260923 19:29:23.076936 2642 ts_tablet_manager.cc:1397] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 19:29:23.076961 2643 ts_tablet_manager.cc:1397] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 19:29:23.076160 2644 ts_tablet_manager.cc:1428] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 19:29:23.077677 2643 raft_consensus.cc:348] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.077759 2642 raft_consensus.cc:348] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.077914 2643 raft_consensus.cc:374] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:23.078002 2642 raft_consensus.cc:374] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:23.078009 2643 raft_consensus.cc:729] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aeb6578f474046943d0466adbeed9e, State: Initialized, Role: FOLLOWER
I20260923 19:29:23.078099 2642 raft_consensus.cc:729] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5119edcb73284b7297435397958a1abd, State: Initialized, Role: FOLLOWER
I20260923 19:29:23.078243 2643 consensus_queue.cc:260] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.078734 2650 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.079103 2650 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:23.078373 2642 consensus_queue.cc:260] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.079833 2642 ts_tablet_manager.cc:1428] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 19:29:23.079632 2648 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.078953 2643 ts_tablet_manager.cc:1428] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 19:29:23.080036 2648 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:23.157900 2650 raft_consensus.cc:482] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:23.158169 2650 raft_consensus.cc:504] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.158906 2650 leader_election.cc:288] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5119edcb73284b7297435397958a1abd (127.0.97.1:41265), d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515)
I20260923 19:29:23.159559 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d34a3469ce4de4b4732f95f5f9e61a" candidate_uuid: "68aeb6578f474046943d0466adbeed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5119edcb73284b7297435397958a1abd" is_pre_election: true
I20260923 19:29:23.159577 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d34a3469ce4de4b4732f95f5f9e61a" candidate_uuid: "68aeb6578f474046943d0466adbeed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684" is_pre_election: true
I20260923 19:29:23.159843 2578 raft_consensus.cc:2454] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aeb6578f474046943d0466adbeed9e in term 0.
I20260923 19:29:23.159837 2295 raft_consensus.cc:2454] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aeb6578f474046943d0466adbeed9e in term 0.
I20260923 19:29:23.160276 2365 leader_election.cc:302] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5119edcb73284b7297435397958a1abd, 68aeb6578f474046943d0466adbeed9e; no voters:
I20260923 19:29:23.160602 2650 raft_consensus.cc:2781] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:23.160691 2650 raft_consensus.cc:482] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:23.160804 2650 raft_consensus.cc:3037] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:23.163136 2650 raft_consensus.cc:504] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.163689 2650 leader_election.cc:288] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [CANDIDATE]: Term 1 election: Requested vote from peers 5119edcb73284b7297435397958a1abd (127.0.97.1:41265), d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515)
I20260923 19:29:23.164189 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d34a3469ce4de4b4732f95f5f9e61a" candidate_uuid: "68aeb6578f474046943d0466adbeed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5119edcb73284b7297435397958a1abd"
I20260923 19:29:23.164306 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d34a3469ce4de4b4732f95f5f9e61a" candidate_uuid: "68aeb6578f474046943d0466adbeed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684"
I20260923 19:29:23.164460 2295 raft_consensus.cc:3037] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:23.164562 2578 raft_consensus.cc:3037] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:23.167555 2578 raft_consensus.cc:2454] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aeb6578f474046943d0466adbeed9e in term 1.
I20260923 19:29:23.167555 2295 raft_consensus.cc:2454] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aeb6578f474046943d0466adbeed9e in term 1.
I20260923 19:29:23.168026 2367 leader_election.cc:302] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aeb6578f474046943d0466adbeed9e, d3243e03e96e4dde814f8c392e9d0684; no voters:
I20260923 19:29:23.168327 2650 raft_consensus.cc:2781] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:23.168551 2650 raft_consensus.cc:686] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [term 1 LEADER]: Becoming Leader. State: Replica: 68aeb6578f474046943d0466adbeed9e, State: Running, Role: LEADER
I20260923 19:29:23.168802 2650 consensus_queue.cc:237] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.170588 2653 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: TxnStatusTablet state changed. Reason: New leader 68aeb6578f474046943d0466adbeed9e. Latest consensus state: current_term: 1 leader_uuid: "68aeb6578f474046943d0466adbeed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.171043 2653 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: This TxnStatusTablet replica's current role is: LEADER
I20260923 19:29:23.171836 2665 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 19:29:23.172132 2665 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 19:29:23.172360 2138 catalog_manager.cc:5495] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 68aeb6578f474046943d0466adbeed9e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "68aeb6578f474046943d0466adbeed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } health_report { overall_health: UNKNOWN } } }
I20260923 19:29:23.214915 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.207801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38407 (local address 127.0.0.1:34594)
0923 19:29:23.208046 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:23.208064 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:23.208172 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:23.208853 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:23.208858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:23.208872 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:29:23.209081 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.209094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.210601 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.210605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.211626 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.211638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.212710 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.212716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.212952 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.213030 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.214351 (+ 1321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.214370 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 19:29:23.214446 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 19:29:23.214572 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 19:29:23.214952 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.207900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.0.97.2:38407)
0923 19:29:23.208142 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:23.208149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:23.208178 (+ 29us) server_negotiation.cc:407] Connection header received
0923 19:29:23.208398 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:23.208404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:23.208606 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:23.208806 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:23.209220 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.210453 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.211776 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.212480 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.212641 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.213284 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.214193 (+ 909us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.214389 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:23.214483 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 19:29:23.214583 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 19:29:23.223107 2295 raft_consensus.cc:1261] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Refusing update from remote peer 68aeb6578f474046943d0466adbeed9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:23.223100 2578 raft_consensus.cc:1261] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Refusing update from remote peer 68aeb6578f474046943d0466adbeed9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:23.224120 2650 consensus_queue.cc:1035] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:23.224435 2653 consensus_queue.cc:1035] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:23.233583 2648 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: TxnStatusTablet state changed. Reason: New leader 68aeb6578f474046943d0466adbeed9e. Latest consensus state: current_term: 1 leader_uuid: "68aeb6578f474046943d0466adbeed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.233866 2648 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:23.238322 2669 mvcc.cc:205] Tried to move back new op lower bound from 7332625462151430144 to 7332625461941841920. Current Snapshot: MvccSnapshot[applied={T|T < 7332625462151430144}]
I20260923 19:29:23.238157 2649 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: TxnStatusTablet state changed. Reason: New leader 68aeb6578f474046943d0466adbeed9e. Latest consensus state: current_term: 1 leader_uuid: "68aeb6578f474046943d0466adbeed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.238443 2649 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:23.242162 2648 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68aeb6578f474046943d0466adbeed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.242419 2648 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P 5119edcb73284b7297435397958a1abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:23.246826 2649 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68aeb6578f474046943d0466adbeed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.247046 2649 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P d3243e03e96e4dde814f8c392e9d0684: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:29:23.249353 2653 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68aeb6578f474046943d0466adbeed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.249706 2653 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: This TxnStatusTablet replica's current role is: LEADER
I20260923 19:29:23.255391 2650 tablet_replica.cc:439] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68aeb6578f474046943d0466adbeed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } } }
I20260923 19:29:23.255846 2650 tablet_replica.cc:442] T e0d34a3469ce4de4b4732f95f5f9e61a P 68aeb6578f474046943d0466adbeed9e: This TxnStatusTablet replica's current role is: LEADER
I20260923 19:29:23.268196 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.262222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.0.97.2:38407)
0923 19:29:23.262464 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:23.262472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:23.262498 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:29:23.262724 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:23.262729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:23.262830 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:23.262944 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:23.263332 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.264626 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.266056 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.266780 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.266916 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.267336 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.267513 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.267604 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:23.267747 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 19:29:23.267838 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 19:29:23.286361 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.280592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37115 (local address 127.0.97.2:38407)
0923 19:29:23.280760 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:23.280765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:23.281583 (+ 818us) server_negotiation.cc:407] Connection header received
0923 19:29:23.281741 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:23.281745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:23.281799 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:23.281872 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:23.282430 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.283188 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.284642 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.285078 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.285180 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.285627 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.285745 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.285812 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:23.285935 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 19:29:23.286005 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 19:29:23.286453 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.280703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38407 (local address 127.0.97.2:37115)
0923 19:29:23.281463 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:23.281480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:23.281625 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:23.281904 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:23.281908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:23.281918 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:29:23.282246 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.282254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.283324 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.283332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.284515 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.284530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.285255 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.285260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.285389 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.285441 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.285830 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.285840 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 19:29:23.285904 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 19:29:23.286038 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":457,"threads_started":1}
I20260923 19:29:23.295759 2578 raft_consensus.cc:1261] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Refusing update from remote peer 68aeb6578f474046943d0466adbeed9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:23.295850 2295 raft_consensus.cc:1261] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Refusing update from remote peer 68aeb6578f474046943d0466adbeed9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:23.296346 2653 consensus_queue.cc:1035] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:23.296736 2676 consensus_queue.cc:1035] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:23.308104 2670 mvcc.cc:205] Tried to move back new op lower bound from 7332625462454337536 to 7332625461080072192. Current Snapshot: MvccSnapshot[applied={T|T < 7332625462454337536}]
I20260923 19:29:23.309671 2668 mvcc.cc:205] Tried to move back new op lower bound from 7332625462454337536 to 7332625461080072192. Current Snapshot: MvccSnapshot[applied={T|T < 7332625462454337536}]
I20260923 19:29:23.363281 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:23.363547 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:23.367285 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1: Sending DeleteTablet for 3 replicas of tablet 4e772f14204a4e3d888d173fe3917777
I20260923 19:29:23.368389 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e772f14204a4e3d888d173fe3917777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34592
I20260923 19:29:23.368366 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e772f14204a4e3d888d173fe3917777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38822
I20260923 19:29:23.369580 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e772f14204a4e3d888d173fe3917777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40794
I20260923 19:29:23.369766 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:29:23.370479 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:23.371991 2684 tablet_replica.cc:330] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: stopping tablet replica
I20260923 19:29:23.372200 2686 tablet_replica.cc:330] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: stopping tablet replica
I20260923 19:29:23.372444 2684 raft_consensus.cc:2229] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:29:23.372663 2686 raft_consensus.cc:2229] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:23.373059 2686 raft_consensus.cc:2258] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:23.373085 2684 raft_consensus.cc:2258] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:23.376010 2686 ts_tablet_manager.cc:1905] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:23.379267 2684 ts_tablet_manager.cc:1905] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:23.380445 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 151323cff5ef48ee9134199365b22008 (DEFAULT_TABLE table=test-table [id=bb55aa4e52ed4143b93f09bff3d64ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:23.380918 2685 tablet_replica.cc:330] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: stopping tablet replica
I20260923 19:29:23.380995 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151323cff5ef48ee9134199365b22008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:23.381419 2685 raft_consensus.cc:2229] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:23.381826 2685 raft_consensus.cc:2258] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:23.384816 2686 ts_tablet_manager.cc:1918] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:29:23.386577 2552 tablet_service.cc:1467] Processing CreateTablet for tablet 151323cff5ef48ee9134199365b22008 (DEFAULT_TABLE table=test-table [id=bb55aa4e52ed4143b93f09bff3d64ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:23.386895 2685 ts_tablet_manager.cc:1905] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:23.387107 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151323cff5ef48ee9134199365b22008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:23.387315 2686 log.cc:1196] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/4e772f14204a4e3d888d173fe3917777
I20260923 19:29:23.387840 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 151323cff5ef48ee9134199365b22008 (DEFAULT_TABLE table=test-table [id=bb55aa4e52ed4143b93f09bff3d64ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:23.391984 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151323cff5ef48ee9134199365b22008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:23.392289 2686 ts_tablet_manager.cc:1939] T 4e772f14204a4e3d888d173fe3917777 P 5119edcb73284b7297435397958a1abd: Deleting consensus metadata
I20260923 19:29:23.393394 2684 ts_tablet_manager.cc:1918] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:29:23.393558 2684 log.cc:1196] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/4e772f14204a4e3d888d173fe3917777
I20260923 19:29:23.394088 2684 ts_tablet_manager.cc:1939] T 4e772f14204a4e3d888d173fe3917777 P 68aeb6578f474046943d0466adbeed9e: Deleting consensus metadata
I20260923 19:29:23.394557 2643 tablet_bootstrap.cc:492] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: Bootstrap starting.
I20260923 19:29:23.396477 2643 tablet_bootstrap.cc:654] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:23.397796 2685 ts_tablet_manager.cc:1918] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:29:23.397943 2685 log.cc:1196] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/4e772f14204a4e3d888d173fe3917777
I20260923 19:29:23.398068 2124 catalog_manager.cc:4841] TS 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407): tablet 4e772f14204a4e3d888d173fe3917777 (table test-table [id=4f8c8e5f2a5e42a7b0c38877ed5fa5ba]) successfully deleted
I20260923 19:29:23.398516 2122 catalog_manager.cc:4841] TS 5119edcb73284b7297435397958a1abd (127.0.97.1:41265): tablet 4e772f14204a4e3d888d173fe3917777 (table test-table [id=4f8c8e5f2a5e42a7b0c38877ed5fa5ba]) successfully deleted
I20260923 19:29:23.398603 2685 ts_tablet_manager.cc:1939] T 4e772f14204a4e3d888d173fe3917777 P d3243e03e96e4dde814f8c392e9d0684: Deleting consensus metadata
I20260923 19:29:23.399995 2124 catalog_manager.cc:4841] TS d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515): tablet 4e772f14204a4e3d888d173fe3917777 (table test-table [id=4f8c8e5f2a5e42a7b0c38877ed5fa5ba]) successfully deleted
I20260923 19:29:23.403623 2643 tablet_bootstrap.cc:492] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: No bootstrap required, opened a new log
I20260923 19:29:23.403740 2642 tablet_bootstrap.cc:492] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: Bootstrap starting.
I20260923 19:29:23.403874 2643 ts_tablet_manager.cc:1397] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260923 19:29:23.404716 2643 raft_consensus.cc:348] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.404970 2643 raft_consensus.cc:374] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:23.405052 2643 raft_consensus.cc:729] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aeb6578f474046943d0466adbeed9e, State: Initialized, Role: FOLLOWER
I20260923 19:29:23.405344 2643 consensus_queue.cc:260] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.405620 2642 tablet_bootstrap.cc:654] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:23.405929 2643 ts_tablet_manager.cc:1428] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 19:29:23.406410 2644 tablet_bootstrap.cc:492] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: Bootstrap starting.
I20260923 19:29:23.408077 2644 tablet_bootstrap.cc:654] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:23.408751 2642 tablet_bootstrap.cc:492] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: No bootstrap required, opened a new log
I20260923 19:29:23.408902 2642 ts_tablet_manager.cc:1397] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 19:29:23.409623 2642 raft_consensus.cc:348] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.409858 2642 raft_consensus.cc:374] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:23.409931 2642 raft_consensus.cc:729] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5119edcb73284b7297435397958a1abd, State: Initialized, Role: FOLLOWER
I20260923 19:29:23.410199 2642 consensus_queue.cc:260] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.411204 2642 ts_tablet_manager.cc:1428] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 19:29:23.412915 2644 tablet_bootstrap.cc:492] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: No bootstrap required, opened a new log
I20260923 19:29:23.413051 2644 ts_tablet_manager.cc:1397] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260923 19:29:23.413645 2644 raft_consensus.cc:348] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.413842 2644 raft_consensus.cc:374] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:23.413921 2644 raft_consensus.cc:729] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3243e03e96e4dde814f8c392e9d0684, State: Initialized, Role: FOLLOWER
I20260923 19:29:23.414217 2644 consensus_queue.cc:260] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.414880 2644 ts_tablet_manager.cc:1428] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 19:29:23.483968 2648 raft_consensus.cc:482] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:23.484779 2648 raft_consensus.cc:504] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.486246 2648 leader_election.cc:288] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407), d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515)
I20260923 19:29:23.495637 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.486702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37321 (local address 127.0.97.2:38407)
0923 19:29:23.486944 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:23.486951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:23.487157 (+ 206us) server_negotiation.cc:407] Connection header received
0923 19:29:23.487460 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:23.487467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:23.487550 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:23.487623 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:23.488437 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.489707 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.492406 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.493943 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.494085 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.494209 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:23.495148 (+ 939us) server_negotiation.cc:299] Negotiation successful
0923 19:29:23.495311 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 19:29:23.497697 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.486548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38407 (local address 127.0.97.1:37321)
0923 19:29:23.487041 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:23.487062 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:23.487261 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:23.487887 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:23.487892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:23.487919 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:23.488216 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.488238 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.489910 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.489916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.492189 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.492206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.494708 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.494715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.494923 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.494962 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 19:29:23.497112 (+ 2150us) client_negotiation.cc:240] Negotiation successful
0923 19:29:23.497284 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":268,"threads_started":1}
I20260923 19:29:23.499776 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151323cff5ef48ee9134199365b22008" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aeb6578f474046943d0466adbeed9e" is_pre_election: true
I20260923 19:29:23.500341 2439 raft_consensus.cc:2454] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 0.
I20260923 19:29:23.501220 2227 leader_election.cc:302] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5119edcb73284b7297435397958a1abd, 68aeb6578f474046943d0466adbeed9e; no voters:
I20260923 19:29:23.501797 2648 raft_consensus.cc:2781] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:23.501932 2648 raft_consensus.cc:482] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:23.502031 2648 raft_consensus.cc:3037] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:23.504647 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.487197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33515 (local address 127.0.97.1:44273)
0923 19:29:23.497983 (+ 10786us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:23.498006 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:23.498139 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:23.498963 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:23.498967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:23.498991 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:23.499158 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.499168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.500333 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.500337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.501688 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.501704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.503880 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.503886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.504100 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.504134 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:29:23.504233 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 19:29:23.504366 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10629,"thread_start_us":11491,"threads_started":1}
I20260923 19:29:23.504813 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.487375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44273 (local address 127.0.97.3:33515)
0923 19:29:23.498633 (+ 11258us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:23.498640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:23.498661 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:29:23.498744 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:23.498748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:23.498818 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:23.499287 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:23.499412 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.500176 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.502248 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.503667 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.503848 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.503967 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:23.504295 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 19:29:23.504480 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11127,"thread_start_us":11013,"threads_started":1}
I20260923 19:29:23.504987 2648 raft_consensus.cc:504] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.505656 2648 leader_election.cc:288] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 election: Requested vote from peers 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407), d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515)
I20260923 19:29:23.505709 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151323cff5ef48ee9134199365b22008" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684" is_pre_election: true
I20260923 19:29:23.505998 2578 raft_consensus.cc:2454] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 0.
I20260923 19:29:23.506067 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151323cff5ef48ee9134199365b22008" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aeb6578f474046943d0466adbeed9e"
I20260923 19:29:23.506322 2439 raft_consensus.cc:3037] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:23.506402 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151323cff5ef48ee9134199365b22008" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684"
I20260923 19:29:23.506572 2577 raft_consensus.cc:3037] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:23.509189 2439 raft_consensus.cc:2454] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 1.
I20260923 19:29:23.509610 2227 leader_election.cc:302] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5119edcb73284b7297435397958a1abd, 68aeb6578f474046943d0466adbeed9e; no voters:
I20260923 19:29:23.509828 2577 raft_consensus.cc:2454] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 1.
I20260923 19:29:23.509874 2648 raft_consensus.cc:2781] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:23.518608 2648 raft_consensus.cc:686] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 1 LEADER]: Becoming Leader. State: Replica: 5119edcb73284b7297435397958a1abd, State: Running, Role: LEADER
I20260923 19:29:23.519037 2648 consensus_queue.cc:237] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:23.523465 2137 catalog_manager.cc:5495] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5119edcb73284b7297435397958a1abd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5119edcb73284b7297435397958a1abd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } health_report { overall_health: UNKNOWN } } }
I20260923 19:29:23.551545 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.546114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.0.97.1:41265)
0923 19:29:23.546660 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:23.546664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:23.546680 (+ 16us) server_negotiation.cc:407] Connection header received
0923 19:29:23.546754 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:23.546757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:23.546824 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:23.546977 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:23.547355 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.548138 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.549369 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.549761 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.549862 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.550318 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.550941 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.551080 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:23.551129 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 19:29:23.551203 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":322,"threads_started":1}
I20260923 19:29:23.569563 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.562837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38407 (local address 127.0.97.1:34725)
0923 19:29:23.563538 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:23.563560 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:23.563761 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:23.564248 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:23.564252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:23.564262 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:29:23.564546 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.564554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.566068 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.566072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.567268 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.567280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.568323 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.568328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.568466 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.568523 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.569045 (+ 522us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.569056 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 19:29:23.569144 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 19:29:23.569258 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":386,"threads_started":1}
I20260923 19:29:23.569665 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.562850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34725 (local address 127.0.97.2:38407)
0923 19:29:23.563013 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:23.563017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:23.563687 (+ 670us) server_negotiation.cc:407] Connection header received
0923 19:29:23.563946 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:23.563952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:23.564087 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:23.564232 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:23.564707 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.565893 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.567423 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.568094 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.568253 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.568724 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.568909 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.569006 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:23.569192 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 19:29:23.569288 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 19:29:23.578527 2577 raft_consensus.cc:1261] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Refusing update from remote peer 5119edcb73284b7297435397958a1abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:23.578671 2439 raft_consensus.cc:1261] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Refusing update from remote peer 5119edcb73284b7297435397958a1abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:23.579200 2648 consensus_queue.cc:1035] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:23.579612 2693 consensus_queue.cc:1035] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:23.607463 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.601938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47371 (local address 127.0.97.1:41265)
0923 19:29:23.602109 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:23.602116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:23.602346 (+ 230us) server_negotiation.cc:407] Connection header received
0923 19:29:23.602422 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:23.602428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:23.602513 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:23.602587 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:23.602938 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.603951 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.605383 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.605941 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.606044 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.606608 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.606779 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.606894 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:23.607102 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 19:29:23.607182 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 19:29:23.607595 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:23.601783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41265 (local address 127.0.97.2:47371)
0923 19:29:23.602118 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:23.602135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:23.602259 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:23.602604 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:23.602608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:23.602619 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:29:23.602814 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.602824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.604087 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.604092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.605260 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:23.605271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:23.606151 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:23.606155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:23.606356 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:23.606402 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:29:23.606959 (+ 557us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:29:23.606973 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 19:29:23.607059 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 19:29:23.607239 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
W20260923 19:29:23.608738 2669 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 19:29:25.225430 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:25.225716 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:25.230969 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1: Sending DeleteTablet for 3 replicas of tablet 151323cff5ef48ee9134199365b22008
I20260923 19:29:25.231942 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 151323cff5ef48ee9134199365b22008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40794
I20260923 19:29:25.232015 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 151323cff5ef48ee9134199365b22008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34592
I20260923 19:29:25.232000 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet 151323cff5ef48ee9134199365b22008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38822
I20260923 19:29:25.244374 2724 tablet_replica.cc:330] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: stopping tablet replica
I20260923 19:29:25.244719 2724 raft_consensus.cc:2229] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:25.245061 2724 raft_consensus.cc:2258] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:25.245685 2725 tablet_replica.cc:330] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: stopping tablet replica
I20260923 19:29:25.245908 2725 raft_consensus.cc:2229] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:25.246173 2725 raft_consensus.cc:2258] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:25.246480 2723 tablet_replica.cc:330] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: stopping tablet replica
I20260923 19:29:25.246698 2723 raft_consensus.cc:2229] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:29:25.247372 2725 ts_tablet_manager.cc:1905] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:25.247380 2724 ts_tablet_manager.cc:1905] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:25.247375 2723 raft_consensus.cc:2258] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:25.248274 2723 ts_tablet_manager.cc:1905] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:25.252542 2725 ts_tablet_manager.cc:1918] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 19:29:25.252660 2725 log.cc:1196] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/151323cff5ef48ee9134199365b22008
I20260923 19:29:25.253062 2725 ts_tablet_manager.cc:1939] T 151323cff5ef48ee9134199365b22008 P d3243e03e96e4dde814f8c392e9d0684: Deleting consensus metadata
I20260923 19:29:25.254232 2124 catalog_manager.cc:4841] TS d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515): tablet 151323cff5ef48ee9134199365b22008 (table test-table [id=bb55aa4e52ed4143b93f09bff3d64ca5]) successfully deleted
I20260923 19:29:25.254592 2724 ts_tablet_manager.cc:1918] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 19:29:25.254727 2724 log.cc:1196] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/151323cff5ef48ee9134199365b22008
I20260923 19:29:25.255175 2724 ts_tablet_manager.cc:1939] T 151323cff5ef48ee9134199365b22008 P 68aeb6578f474046943d0466adbeed9e: Deleting consensus metadata
I20260923 19:29:25.256026 2723 ts_tablet_manager.cc:1918] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 19:29:25.256191 2723 log.cc:1196] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/151323cff5ef48ee9134199365b22008
I20260923 19:29:25.256541 2723 ts_tablet_manager.cc:1939] T 151323cff5ef48ee9134199365b22008 P 5119edcb73284b7297435397958a1abd: Deleting consensus metadata
I20260923 19:29:25.256543 2124 catalog_manager.cc:4841] TS 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407): tablet 151323cff5ef48ee9134199365b22008 (table test-table [id=bb55aa4e52ed4143b93f09bff3d64ca5]) successfully deleted
I20260923 19:29:25.257524 2122 catalog_manager.cc:4841] TS 5119edcb73284b7297435397958a1abd (127.0.97.1:41265): tablet 151323cff5ef48ee9134199365b22008 (table test-table [id=bb55aa4e52ed4143b93f09bff3d64ca5]) successfully deleted
Sep 23 19:29:25 dist-test-slave-kxfb krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191765, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 19:29:25 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191765, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:25.298177 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:25.283625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.97.62:38485)
0923 19:29:25.284287 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:25.284294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:25.284325 (+ 31us) server_negotiation.cc:407] Connection header received
0923 19:29:25.284556 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:25.284560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:25.284625 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:25.284724 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:25.286073 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:25.286932 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:25.288191 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:25.288581 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:25.288687 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:25.292901 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:25.292907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:25.292910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:25.292944 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:25.295605 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:25.296241 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:25.296247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:25.296250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:25.296371 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:25.296730 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:25.296734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:25.296738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:25.297027 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:25.297260 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:25.297558 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 19:29:25.297801 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":378,"threads_started":1}
I20260923 19:29:25.301774 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:29:25.302497 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:25.309372 2275 tablet_service.cc:1467] Processing CreateTablet for tablet a559ddbc171f44c7a61e62eb5ded6823 (DEFAULT_TABLE table=test-table [id=d6d79428e51340dabd9aa238b3e8256e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:25.309513 2414 tablet_service.cc:1467] Processing CreateTablet for tablet a559ddbc171f44c7a61e62eb5ded6823 (DEFAULT_TABLE table=test-table [id=d6d79428e51340dabd9aa238b3e8256e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:25.309614 2552 tablet_service.cc:1467] Processing CreateTablet for tablet a559ddbc171f44c7a61e62eb5ded6823 (DEFAULT_TABLE table=test-table [id=d6d79428e51340dabd9aa238b3e8256e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:25.309892 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a559ddbc171f44c7a61e62eb5ded6823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:25.309887 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a559ddbc171f44c7a61e62eb5ded6823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:25.310074 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a559ddbc171f44c7a61e62eb5ded6823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:25.317137 2734 tablet_bootstrap.cc:492] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: Bootstrap starting.
I20260923 19:29:25.317845 2736 tablet_bootstrap.cc:492] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: Bootstrap starting.
I20260923 19:29:25.318630 2735 tablet_bootstrap.cc:492] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: Bootstrap starting.
I20260923 19:29:25.319342 2734 tablet_bootstrap.cc:654] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:25.319550 2736 tablet_bootstrap.cc:654] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:25.320966 2735 tablet_bootstrap.cc:654] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:25.322846 2734 tablet_bootstrap.cc:492] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: No bootstrap required, opened a new log
I20260923 19:29:25.323122 2734 ts_tablet_manager.cc:1397] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260923 19:29:25.323292 2735 tablet_bootstrap.cc:492] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: No bootstrap required, opened a new log
I20260923 19:29:25.323378 2736 tablet_bootstrap.cc:492] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: No bootstrap required, opened a new log
I20260923 19:29:25.323446 2735 ts_tablet_manager.cc:1397] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 19:29:25.323567 2736 ts_tablet_manager.cc:1397] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 19:29:25.323922 2734 raft_consensus.cc:348] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.324179 2734 raft_consensus.cc:374] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:25.324272 2734 raft_consensus.cc:729] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3243e03e96e4dde814f8c392e9d0684, State: Initialized, Role: FOLLOWER
I20260923 19:29:25.324237 2735 raft_consensus.cc:348] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.324479 2735 raft_consensus.cc:374] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:25.324388 2736 raft_consensus.cc:348] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.324558 2735 raft_consensus.cc:729] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aeb6578f474046943d0466adbeed9e, State: Initialized, Role: FOLLOWER
I20260923 19:29:25.324617 2736 raft_consensus.cc:374] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:25.324568 2734 consensus_queue.cc:260] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.324702 2736 raft_consensus.cc:729] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5119edcb73284b7297435397958a1abd, State: Initialized, Role: FOLLOWER
I20260923 19:29:25.324862 2735 consensus_queue.cc:260] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.324915 2736 consensus_queue.cc:260] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.325469 2736 ts_tablet_manager.cc:1428] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 19:29:25.325965 2734 ts_tablet_manager.cc:1428] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 19:29:25.327116 2735 ts_tablet_manager.cc:1428] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 19:29:25.388676 2718 raft_consensus.cc:482] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:25.388971 2718 raft_consensus.cc:504] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.389772 2718 leader_election.cc:288] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407), d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515)
I20260923 19:29:25.390538 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a559ddbc171f44c7a61e62eb5ded6823" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aeb6578f474046943d0466adbeed9e" is_pre_election: true
I20260923 19:29:25.390615 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a559ddbc171f44c7a61e62eb5ded6823" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684" is_pre_election: true
I20260923 19:29:25.390798 2438 raft_consensus.cc:2454] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 0.
I20260923 19:29:25.390870 2578 raft_consensus.cc:2454] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 0.
I20260923 19:29:25.391258 2227 leader_election.cc:302] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5119edcb73284b7297435397958a1abd, 68aeb6578f474046943d0466adbeed9e; no voters:
I20260923 19:29:25.391530 2718 raft_consensus.cc:2781] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:25.391631 2718 raft_consensus.cc:482] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:25.391717 2718 raft_consensus.cc:3037] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:25.394831 2718 raft_consensus.cc:504] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.395439 2718 leader_election.cc:288] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 election: Requested vote from peers 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407), d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515)
I20260923 19:29:25.395959 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a559ddbc171f44c7a61e62eb5ded6823" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aeb6578f474046943d0466adbeed9e"
I20260923 19:29:25.396013 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a559ddbc171f44c7a61e62eb5ded6823" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684"
I20260923 19:29:25.396164 2438 raft_consensus.cc:3037] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:25.396214 2578 raft_consensus.cc:3037] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:25.398339 2438 raft_consensus.cc:2454] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 1.
I20260923 19:29:25.398344 2578 raft_consensus.cc:2454] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 1.
I20260923 19:29:25.398732 2227 leader_election.cc:302] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5119edcb73284b7297435397958a1abd, d3243e03e96e4dde814f8c392e9d0684; no voters:
I20260923 19:29:25.398988 2718 raft_consensus.cc:2781] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:25.399122 2718 raft_consensus.cc:686] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 1 LEADER]: Becoming Leader. State: Replica: 5119edcb73284b7297435397958a1abd, State: Running, Role: LEADER
I20260923 19:29:25.399391 2718 consensus_queue.cc:237] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.402253 2136 catalog_manager.cc:5495] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5119edcb73284b7297435397958a1abd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5119edcb73284b7297435397958a1abd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } health_report { overall_health: UNKNOWN } } }
I20260923 19:29:25.433658 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:25.427983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.0.97.1:41265)
0923 19:29:25.428595 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:25.428599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:25.428620 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:29:25.428684 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:25.428688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:25.428756 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:25.428865 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:25.429356 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:25.430201 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:25.431726 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:25.432200 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:25.432333 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:25.432799 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:25.432931 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:25.433025 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:25.433199 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 19:29:25.433284 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":326,"threads_started":1}
I20260923 19:29:25.443796 2578 raft_consensus.cc:1261] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Refusing update from remote peer 5119edcb73284b7297435397958a1abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:25.443840 2438 raft_consensus.cc:1261] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Refusing update from remote peer 5119edcb73284b7297435397958a1abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:25.444424 2718 consensus_queue.cc:1035] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:25.444715 2693 consensus_queue.cc:1035] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:25.491530 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:25.491806 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:25.495690 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1: Sending DeleteTablet for 3 replicas of tablet a559ddbc171f44c7a61e62eb5ded6823
I20260923 19:29:25.496544 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet a559ddbc171f44c7a61e62eb5ded6823 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40794
I20260923 19:29:25.496675 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet a559ddbc171f44c7a61e62eb5ded6823 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38822
I20260923 19:29:25.496873 2723 tablet_replica.cc:330] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: stopping tablet replica
I20260923 19:29:25.496971 2725 tablet_replica.cc:330] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: stopping tablet replica
I20260923 19:29:25.496968 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet a559ddbc171f44c7a61e62eb5ded6823 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34592
I20260923 19:29:25.497102 2723 raft_consensus.cc:2229] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:29:25.497180 2725 raft_consensus.cc:2229] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:25.497265 2724 tablet_replica.cc:330] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: stopping tablet replica
I20260923 19:29:25.497392 2725 raft_consensus.cc:2258] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:25.497443 2723 raft_consensus.cc:2258] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:25.497465 2724 raft_consensus.cc:2229] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:25.497640 2724 raft_consensus.cc:2258] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:25.498417 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 19:29:25.498740 2725 ts_tablet_manager.cc:1905] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:25.498740 2724 ts_tablet_manager.cc:1905] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:25.498860 2723 ts_tablet_manager.cc:1905] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 19:29:25.499117 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:25.504703 2723 ts_tablet_manager.cc:1918] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:29:25.504870 2723 log.cc:1196] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/a559ddbc171f44c7a61e62eb5ded6823
I20260923 19:29:25.505294 2723 ts_tablet_manager.cc:1939] T a559ddbc171f44c7a61e62eb5ded6823 P 5119edcb73284b7297435397958a1abd: Deleting consensus metadata
I20260923 19:29:25.505393 2725 ts_tablet_manager.cc:1918] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:29:25.505555 2725 log.cc:1196] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/a559ddbc171f44c7a61e62eb5ded6823
I20260923 19:29:25.505961 2725 ts_tablet_manager.cc:1939] T a559ddbc171f44c7a61e62eb5ded6823 P d3243e03e96e4dde814f8c392e9d0684: Deleting consensus metadata
I20260923 19:29:25.506606 2724 ts_tablet_manager.cc:1918] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:29:25.506692 2122 catalog_manager.cc:4841] TS 5119edcb73284b7297435397958a1abd (127.0.97.1:41265): tablet a559ddbc171f44c7a61e62eb5ded6823 (table test-table [id=d6d79428e51340dabd9aa238b3e8256e]) successfully deleted
I20260923 19:29:25.506856 2724 log.cc:1196] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/a559ddbc171f44c7a61e62eb5ded6823
I20260923 19:29:25.507102 2124 catalog_manager.cc:4841] TS d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515): tablet a559ddbc171f44c7a61e62eb5ded6823 (table test-table [id=d6d79428e51340dabd9aa238b3e8256e]) successfully deleted
I20260923 19:29:25.507584 2724 ts_tablet_manager.cc:1939] T a559ddbc171f44c7a61e62eb5ded6823 P 68aeb6578f474046943d0466adbeed9e: Deleting consensus metadata
I20260923 19:29:25.508900 2124 catalog_manager.cc:4841] TS 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407): tablet a559ddbc171f44c7a61e62eb5ded6823 (table test-table [id=d6d79428e51340dabd9aa238b3e8256e]) successfully deleted
I20260923 19:29:25.508998 2275 tablet_service.cc:1467] Processing CreateTablet for tablet b4797fd66163415a89bc3aaec183fe95 (DEFAULT_TABLE table=test-table [id=14f6c5a19ea6446aa78fabb641180d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:25.509336 2414 tablet_service.cc:1467] Processing CreateTablet for tablet b4797fd66163415a89bc3aaec183fe95 (DEFAULT_TABLE table=test-table [id=14f6c5a19ea6446aa78fabb641180d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:25.509466 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4797fd66163415a89bc3aaec183fe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:25.509581 2552 tablet_service.cc:1467] Processing CreateTablet for tablet b4797fd66163415a89bc3aaec183fe95 (DEFAULT_TABLE table=test-table [id=14f6c5a19ea6446aa78fabb641180d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:25.509744 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4797fd66163415a89bc3aaec183fe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:25.509989 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4797fd66163415a89bc3aaec183fe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:25.515138 2736 tablet_bootstrap.cc:492] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: Bootstrap starting.
I20260923 19:29:25.516125 2734 tablet_bootstrap.cc:492] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: Bootstrap starting.
I20260923 19:29:25.516618 2735 tablet_bootstrap.cc:492] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: Bootstrap starting.
I20260923 19:29:25.516983 2736 tablet_bootstrap.cc:654] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:25.517483 2734 tablet_bootstrap.cc:654] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:25.518407 2735 tablet_bootstrap.cc:654] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:25.520577 2735 tablet_bootstrap.cc:492] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: No bootstrap required, opened a new log
I20260923 19:29:25.521401 2736 tablet_bootstrap.cc:492] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: No bootstrap required, opened a new log
I20260923 19:29:25.521672 2736 ts_tablet_manager.cc:1397] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 19:29:25.521708 2735 ts_tablet_manager.cc:1397] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 19:29:25.521821 2734 tablet_bootstrap.cc:492] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: No bootstrap required, opened a new log
I20260923 19:29:25.521991 2734 ts_tablet_manager.cc:1397] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260923 19:29:25.522521 2736 raft_consensus.cc:348] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.522797 2736 raft_consensus.cc:374] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:25.522895 2736 raft_consensus.cc:729] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5119edcb73284b7297435397958a1abd, State: Initialized, Role: FOLLOWER
I20260923 19:29:25.522724 2734 raft_consensus.cc:348] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.523043 2734 raft_consensus.cc:374] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:25.523121 2734 raft_consensus.cc:729] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3243e03e96e4dde814f8c392e9d0684, State: Initialized, Role: FOLLOWER
I20260923 19:29:25.523160 2736 consensus_queue.cc:260] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.523281 2735 raft_consensus.cc:348] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.523538 2735 raft_consensus.cc:374] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:25.523622 2735 raft_consensus.cc:729] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aeb6578f474046943d0466adbeed9e, State: Initialized, Role: FOLLOWER
I20260923 19:29:25.523910 2736 ts_tablet_manager.cc:1428] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 19:29:25.523849 2735 consensus_queue.cc:260] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.523411 2734 consensus_queue.cc:260] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.524398 2735 ts_tablet_manager.cc:1428] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260923 19:29:25.525173 2734 ts_tablet_manager.cc:1428] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 19:29:25.630074 2713 raft_consensus.cc:482] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:25.630424 2713 raft_consensus.cc:504] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.631249 2713 leader_election.cc:288] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407), d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515)
I20260923 19:29:25.632014 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4797fd66163415a89bc3aaec183fe95" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aeb6578f474046943d0466adbeed9e" is_pre_election: true
I20260923 19:29:25.632064 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4797fd66163415a89bc3aaec183fe95" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684" is_pre_election: true
I20260923 19:29:25.632305 2438 raft_consensus.cc:2454] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 0.
I20260923 19:29:25.632333 2578 raft_consensus.cc:2454] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 0.
I20260923 19:29:25.632797 2227 leader_election.cc:302] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5119edcb73284b7297435397958a1abd, 68aeb6578f474046943d0466adbeed9e; no voters:
I20260923 19:29:25.633102 2713 raft_consensus.cc:2781] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:25.633208 2713 raft_consensus.cc:482] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:25.633299 2713 raft_consensus.cc:3037] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:25.635632 2713 raft_consensus.cc:504] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.636234 2713 leader_election.cc:288] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 election: Requested vote from peers 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407), d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515)
I20260923 19:29:25.636948 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4797fd66163415a89bc3aaec183fe95" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3243e03e96e4dde814f8c392e9d0684"
I20260923 19:29:25.636978 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4797fd66163415a89bc3aaec183fe95" candidate_uuid: "5119edcb73284b7297435397958a1abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aeb6578f474046943d0466adbeed9e"
I20260923 19:29:25.637185 2578 raft_consensus.cc:3037] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:25.637194 2438 raft_consensus.cc:3037] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:25.640262 2578 raft_consensus.cc:2454] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 1.
I20260923 19:29:25.640278 2438 raft_consensus.cc:2454] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5119edcb73284b7297435397958a1abd in term 1.
I20260923 19:29:25.640754 2227 leader_election.cc:302] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5119edcb73284b7297435397958a1abd, d3243e03e96e4dde814f8c392e9d0684; no voters:
I20260923 19:29:25.641079 2713 raft_consensus.cc:2781] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:25.641233 2713 raft_consensus.cc:686] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 1 LEADER]: Becoming Leader. State: Replica: 5119edcb73284b7297435397958a1abd, State: Running, Role: LEADER
I20260923 19:29:25.641532 2713 consensus_queue.cc:237] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } }
I20260923 19:29:25.644352 2137 catalog_manager.cc:5495] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5119edcb73284b7297435397958a1abd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5119edcb73284b7297435397958a1abd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5119edcb73284b7297435397958a1abd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 } health_report { overall_health: UNKNOWN } } }
I20260923 19:29:25.696275 2578 raft_consensus.cc:1261] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Refusing update from remote peer 5119edcb73284b7297435397958a1abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:25.696382 2438 raft_consensus.cc:1261] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Refusing update from remote peer 5119edcb73284b7297435397958a1abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:25.696916 2718 consensus_queue.cc:1035] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3243e03e96e4dde814f8c392e9d0684" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:25.697218 2713 consensus_queue.cc:1035] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "68aeb6578f474046943d0466adbeed9e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 19:29:25.717555 2706 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 19:29:27.305814 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:27.306115 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:27.310770 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 843f2e85983e45408654a1c3727eeaa1: Sending DeleteTablet for 3 replicas of tablet b4797fd66163415a89bc3aaec183fe95
I20260923 19:29:27.311746 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet b4797fd66163415a89bc3aaec183fe95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40794
I20260923 19:29:27.311750 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet b4797fd66163415a89bc3aaec183fe95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34592
I20260923 19:29:27.311857 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet b4797fd66163415a89bc3aaec183fe95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38822
I20260923 19:29:27.312492 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2209
I20260923 19:29:27.324245 2777 tablet_replica.cc:330] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: stopping tablet replica
I20260923 19:29:27.324568 2777 raft_consensus.cc:2229] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:27.324898 2777 raft_consensus.cc:2258] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:27.325609 2776 tablet_replica.cc:330] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: stopping tablet replica
I20260923 19:29:27.325821 2775 tablet_replica.cc:330] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: stopping tablet replica
I20260923 19:29:27.325841 2776 raft_consensus.cc:2229] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:27.326057 2775 raft_consensus.cc:2229] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:29:27.326112 2776 raft_consensus.cc:2258] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:27.326576 2775 raft_consensus.cc:2258] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:27.327160 2777 ts_tablet_manager.cc:1905] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:27.327636 2776 ts_tablet_manager.cc:1905] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:27.327630 2775 ts_tablet_manager.cc:1905] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:27.334645 2777 ts_tablet_manager.cc:1918] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 19:29:27.334652 2775 ts_tablet_manager.cc:1918] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 19:29:27.334789 2775 log.cc:1196] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/b4797fd66163415a89bc3aaec183fe95
I20260923 19:29:27.334794 2777 log.cc:1196] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/b4797fd66163415a89bc3aaec183fe95
I20260923 19:29:27.335181 2777 ts_tablet_manager.cc:1939] T b4797fd66163415a89bc3aaec183fe95 P d3243e03e96e4dde814f8c392e9d0684: Deleting consensus metadata
I20260923 19:29:27.335207 2775 ts_tablet_manager.cc:1939] T b4797fd66163415a89bc3aaec183fe95 P 5119edcb73284b7297435397958a1abd: Deleting consensus metadata
I20260923 19:29:27.336323 2122 catalog_manager.cc:4841] TS 5119edcb73284b7297435397958a1abd (127.0.97.1:41265): tablet b4797fd66163415a89bc3aaec183fe95 (table test-table [id=14f6c5a19ea6446aa78fabb641180d20]) successfully deleted
I20260923 19:29:27.336323 2124 catalog_manager.cc:4841] TS d3243e03e96e4dde814f8c392e9d0684 (127.0.97.3:33515): tablet b4797fd66163415a89bc3aaec183fe95 (table test-table [id=14f6c5a19ea6446aa78fabb641180d20]) successfully deleted
I20260923 19:29:27.336431 2776 ts_tablet_manager.cc:1918] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 19:29:27.336555 2776 log.cc:1196] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/b4797fd66163415a89bc3aaec183fe95
I20260923 19:29:27.336941 2776 ts_tablet_manager.cc:1939] T b4797fd66163415a89bc3aaec183fe95 P 68aeb6578f474046943d0466adbeed9e: Deleting consensus metadata
I20260923 19:29:27.337921 2124 catalog_manager.cc:4841] TS 68aeb6578f474046943d0466adbeed9e (127.0.97.2:38407): tablet b4797fd66163415a89bc3aaec183fe95 (table test-table [id=14f6c5a19ea6446aa78fabb641180d20]) successfully deleted
Sep 23 19:29:27 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191765, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:27.347579 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:27.314373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.0.97.1:41265)
0923 19:29:27.320312 (+ 5939us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:27.320319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:27.326836 (+ 6517us) server_negotiation.cc:407] Connection header received
0923 19:29:27.327111 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:27.327116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:27.327211 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:27.327338 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:27.329133 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:27.330448 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:27.332302 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:27.333093 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:27.333194 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:27.341693 (+ 8499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:27.341713 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:27.341733 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:27.341777 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:27.344903 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:27.345652 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:27.345659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:27.345662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:27.345806 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:27.346084 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:27.346086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:27.346088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:27.346539 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:27.346794 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:27.347055 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 19:29:27.347241 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5774,"thread_start_us":5622,"threads_started":1}
W20260923 19:29:27.348335 2340 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40828
W20260923 19:29:27.348976 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2209: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:29:27.349172 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 2209
I20260923 19:29:27.370790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2349
Sep 23 19:29:27 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191765, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:29:27.387670 2780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:27.371585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.0.97.2:38407)
0923 19:29:27.372130 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:27.372135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:27.372152 (+ 17us) server_negotiation.cc:407] Connection header received
0923 19:29:27.372215 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:27.372219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:27.372285 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:27.372395 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:27.373851 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:27.375491 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:27.376808 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:27.377240 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:27.377333 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:27.381833 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:27.381852 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:27.381873 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:27.381916 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:27.385036 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:27.385661 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:27.385668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:27.385671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:27.385794 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:27.386112 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:27.386116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:27.386118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:27.386659 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:27.386911 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:27.387139 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 19:29:27.387352 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":297,"threads_started":1}
W20260923 19:29:27.388438 2479 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34624
W20260923 19:29:27.389098 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2349: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:29:27.389277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 2349
I20260923 19:29:27.412364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2488
Sep 23 19:29:27 dist-test-slave-kxfb krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191765, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:29:27.428268 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:27.413143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.0.97.3:33515)
0923 19:29:27.413727 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:27.413731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:27.413747 (+ 16us) server_negotiation.cc:407] Connection header received
0923 19:29:27.413808 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:27.413812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:27.413878 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:27.413991 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:27.415399 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:27.416189 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:27.417431 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:27.417840 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:27.417945 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:27.422314 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:27.422346 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:27.422367 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:27.422405 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:27.425477 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:27.426185 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:27.426192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:27.426195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:27.426379 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:27.426715 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:27.426721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:27.426724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:27.427101 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:27.427400 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:27.427670 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 19:29:27.427905 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":320,"threads_started":1}
W20260923 19:29:27.429072 2618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38834
W20260923 19:29:27.429725 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2488: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:29:27.429891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 2488
I20260923 19:29:27.445780 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2106
I20260923 19:29:27.447196 2178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:27.573624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 2106
2026-09-23T19:29:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8213 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:27 dist-test-slave-kxfb krb5kdc[2785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:27 dist-test-slave-kxfb krb5kdc[2785](info): set up 2 sockets
Sep 23 19:29:27 dist-test-slave-kxfb krb5kdc[2785](info): commencing operation
krb5kdc: starting...
W20260923 19:29:28.578698 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:28 dist-test-slave-kxfb krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:28.779881 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34559
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:29.087788 2798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:29.088069 2798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:29.088176 2798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:29.095353 2798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:29.095467 2798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:29.095515 2798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:29.095602 2798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:29.095666 2798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:29.105382 2798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:29.106673 2798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:29.107981 2798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:29.120149 2803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:29.120169 2806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:29.120157 2804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:29.121148 2798 server_base.cc:1034] running on GCE node
I20260923 19:29:29.122114 2798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:29.123958 2798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:29.125200 2798 hybrid_clock.cc:648] HybridClock initialized: now 1790191769125134 us; error 42 us; skew 500 ppm
I20260923 19:29:29.125742 2798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:29 dist-test-slave-kxfb krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:29.129537 2798 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:29.131281 2798 webserver.cc:460] Webserver started at http://127.0.97.62:42117/ using document root <none> and password file <none>
I20260923 19:29:29.132088 2798 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:29.132210 2798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:29.132547 2798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:29.136008 2798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "f3e9a3d9e53a46219e7765e3da01ebf1"
format_stamp: "Formatted at 2026-09-23 19:29:29 on dist-test-slave-kxfb"
server_key: "e157d75ebf5194599afb876748167acb"
server_key_iv: "5864e67686d699ed0fb91423fba2407f"
server_key_version: "encryptionkey@0"
I20260923 19:29:29.136847 2798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "f3e9a3d9e53a46219e7765e3da01ebf1"
format_stamp: "Formatted at 2026-09-23 19:29:29 on dist-test-slave-kxfb"
server_key: "e157d75ebf5194599afb876748167acb"
server_key_iv: "5864e67686d699ed0fb91423fba2407f"
server_key_version: "encryptionkey@0"
I20260923 19:29:29.142925 2798 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 19:29:29.146912 2813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:29.148559 2798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:29:29.148746 2798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "f3e9a3d9e53a46219e7765e3da01ebf1"
format_stamp: "Formatted at 2026-09-23 19:29:29 on dist-test-slave-kxfb"
server_key: "e157d75ebf5194599afb876748167acb"
server_key_iv: "5864e67686d699ed0fb91423fba2407f"
server_key_version: "encryptionkey@0"
I20260923 19:29:29.148968 2798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:29.167263 2798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:29.171684 2798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:29.171968 2798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:29.191195 2798 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34559
I20260923 19:29:29.191226 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34559 every 8 connection(s)
I20260923 19:29:29.192754 2798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:29.196688 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:29.199404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 2798
I20260923 19:29:29.199613 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:29.200326 388 external_mini_cluster.cc:1428] Setting key cb7dfd74957bbe73b0d1ad4d623c50e1
I20260923 19:29:29.209635 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1: Bootstrap starting.
I20260923 19:29:29.213122 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:29.214083 2865 log.cc:824] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1: Log is configured to *not* fsync() on all Append() calls
Sep 23 19:29:29 dist-test-slave-kxfb krb5kdc[2785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:29.217494 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1: No bootstrap required, opened a new log
I20260923 19:29:29.225721 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:29.202921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.0.97.62:34559)
0923 19:29:29.203787 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:29.203804 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:29.203863 (+ 59us) server_negotiation.cc:407] Connection header received
0923 19:29:29.205614 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:29.205645 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:29.206209 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:29.206680 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:29.208215 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:29.210095 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:29.211566 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:29.212545 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:29.212728 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:29.217389 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:29.217400 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:29.217420 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:29.217463 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:29.221133 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:29.221833 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:29.221841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:29.221850 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:29.222043 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:29.222448 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:29.222451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:29.222453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:29.223118 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:29.223444 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:29.223906 (+ 462us) server_negotiation.cc:299] Negotiation successful
0923 19:29:29.224450 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":349,"threads_started":1}
I20260923 19:29:29.225682 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e9a3d9e53a46219e7765e3da01ebf1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34559 } }
I20260923 19:29:29.226164 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:29.226248 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e9a3d9e53a46219e7765e3da01ebf1, State: Initialized, Role: FOLLOWER
I20260923 19:29:29.226889 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e9a3d9e53a46219e7765e3da01ebf1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34559 } }
I20260923 19:29:29.227164 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:29.227315 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:29.227494 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:29.229933 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e9a3d9e53a46219e7765e3da01ebf1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34559 } }
I20260923 19:29:29.230513 2865 leader_election.cc:302] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3e9a3d9e53a46219e7765e3da01ebf1; no voters:
I20260923 19:29:29.231103 2865 leader_election.cc:288] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:29.231268 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:29.232892 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [term 1 LEADER]: Becoming Leader. State: Replica: f3e9a3d9e53a46219e7765e3da01ebf1, State: Running, Role: LEADER
I20260923 19:29:29.233489 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e9a3d9e53a46219e7765e3da01ebf1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34559 } }
I20260923 19:29:29.233764 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:29.236719 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3e9a3d9e53a46219e7765e3da01ebf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e9a3d9e53a46219e7765e3da01ebf1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34559 } } }
I20260923 19:29:29.237260 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:29.236747 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e9a3d9e53a46219e7765e3da01ebf1. Latest consensus state: current_term: 1 leader_uuid: "f3e9a3d9e53a46219e7765e3da01ebf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e9a3d9e53a46219e7765e3da01ebf1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34559 } } }
I20260923 19:29:29.237418 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:29.239836 2877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:29.244976 2877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:29.254112 2877 catalog_manager.cc:1347] Generated new cluster ID: befda571048547d9b653453c4be2d6c7
I20260923 19:29:29.254249 2877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:29.270650 2877 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:29.272099 2877 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:29.282980 2877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3e9a3d9e53a46219e7765e3da01ebf1: Generated new TSK 0
I20260923 19:29:29.283943 2877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:29:29.334051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34559
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:29.695106 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:29.695458 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:29.695616 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:29.703393 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:29.703564 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:29.703727 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:29.714370 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:29.716193 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:29.717728 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:29.732918 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:29.733021 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:29.733036 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:29.733664 2891 server_base.cc:1034] running on GCE node
I20260923 19:29:29.734591 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:29.735935 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:29.737190 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790191769737133 us; error 53 us; skew 500 ppm
I20260923 19:29:29.737600 2891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:29 dist-test-slave-kxfb krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:29.741199 2891 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:29:29.743038 2891 webserver.cc:460] Webserver started at http://127.0.97.1:38319/ using document root <none> and password file <none>
I20260923 19:29:29.743867 2891 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:29.743983 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:29.744339 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:29.747944 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "534b48dc1c314593acf67713158080b2"
format_stamp: "Formatted at 2026-09-23 19:29:29 on dist-test-slave-kxfb"
server_key: "0e69ff6a2a5bd6cb04ebf558693e1fa4"
server_key_iv: "b55c506c69293cd5bdc51c5e64a0cb3f"
server_key_version: "encryptionkey@0"
I20260923 19:29:29.748816 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "534b48dc1c314593acf67713158080b2"
format_stamp: "Formatted at 2026-09-23 19:29:29 on dist-test-slave-kxfb"
server_key: "0e69ff6a2a5bd6cb04ebf558693e1fa4"
server_key_iv: "b55c506c69293cd5bdc51c5e64a0cb3f"
server_key_version: "encryptionkey@0"
I20260923 19:29:29.754933 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 19:29:29.758900 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:29.760600 2891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:29:29.760772 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "534b48dc1c314593acf67713158080b2"
format_stamp: "Formatted at 2026-09-23 19:29:29 on dist-test-slave-kxfb"
server_key: "0e69ff6a2a5bd6cb04ebf558693e1fa4"
server_key_iv: "b55c506c69293cd5bdc51c5e64a0cb3f"
server_key_version: "encryptionkey@0"
I20260923 19:29:29.760983 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:29.777498 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:29.782150 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:29.782478 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:29.783674 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:29.785269 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:29.785362 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:29.785473 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:29.785530 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:29.818905 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44399
I20260923 19:29:29.818981 3018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44399 every 8 connection(s)
I20260923 19:29:29.820390 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:29.826138 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 2891
I20260923 19:29:29.826363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:29.827054 388 external_mini_cluster.cc:1428] Setting key 2443d5400071fce12ec1df724314358e
Sep 23 19:29:29 dist-test-slave-kxfb krb5kdc[2785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:29.841773 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:29.823354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50307 (local address 127.0.97.62:34559)
0923 19:29:29.823888 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:29.823893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:29.825000 (+ 1107us) server_negotiation.cc:407] Connection header received
0923 19:29:29.826991 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:29.826998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:29.827108 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:29.827272 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:29.828883 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:29.829643 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:29.831214 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:29.831605 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:29.831717 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:29.835671 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:29.835678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:29.835682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:29.835721 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:29.838632 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:29.839437 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:29.839441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:29.839444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:29.839573 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:29.839990 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:29.839995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:29.839998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:29.840329 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:29.840464 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:29.841235 (+ 771us) server_negotiation.cc:299] Negotiation successful
0923 19:29:29.841446 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":275,"threads_started":1}
I20260923 19:29:29.842983 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:29.823998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34559 (local address 127.0.97.1:50307)
0923 19:29:29.824803 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:29.824856 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:29.826689 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:29.827501 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:29.827509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:29.828070 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:29.828689 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:29.828706 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:29.829792 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:29.829795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:29.831080 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:29.831091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:29.831818 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:29.831826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:29.832044 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:29.832758 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:29.832779 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:29.835383 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:29.838778 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:29.838787 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:29.838803 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:29.839305 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:29.839680 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:29.839684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:29.839686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:29.839884 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:29.840437 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:29.840443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:29.840826 (+ 383us) client_negotiation.cc:769] Sending connection context
0923 19:29:29.841074 (+ 248us) client_negotiation.cc:240] Negotiation successful
0923 19:29:29.841541 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":292,"threads_started":1}
I20260923 19:29:29.844749 3019 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34559
I20260923 19:29:29.845188 3019 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:29.846261 3019 heartbeater.cc:511] Master 127.0.97.62:34559 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 19:29:29.848858 2830 ts_manager.cc:194] Registered new tserver with Master: 534b48dc1c314593acf67713158080b2 (127.0.97.1:44399)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 19:29:29.850924 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:29:29.873317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34559
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:30.202847 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:30.203142 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:30.203254 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:30.210659 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:30.210779 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:30.210950 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:30.220664 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:30.221987 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:30.223234 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:30.236500 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:30.236426 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:30.236773 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:30.238054 3025 server_base.cc:1034] running on GCE node
I20260923 19:29:30.238864 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:30.240024 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:30.241273 3025 hybrid_clock.cc:648] HybridClock initialized: now 1790191770241209 us; error 45 us; skew 500 ppm
I20260923 19:29:30.241671 3025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:30 dist-test-slave-kxfb krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:30.245405 3025 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:29:30.247284 3025 webserver.cc:460] Webserver started at http://127.0.97.2:33619/ using document root <none> and password file <none>
I20260923 19:29:30.248136 3025 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:30.248282 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:30.248646 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:30.252292 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "7bcf0a64443147acac9235e445a9f8e6"
format_stamp: "Formatted at 2026-09-23 19:29:30 on dist-test-slave-kxfb"
server_key: "2d64e3653448eb808bc12f07b9fb9ae0"
server_key_iv: "8fac24571d7f34839adb66956753146f"
server_key_version: "encryptionkey@0"
I20260923 19:29:30.253284 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7bcf0a64443147acac9235e445a9f8e6"
format_stamp: "Formatted at 2026-09-23 19:29:30 on dist-test-slave-kxfb"
server_key: "2d64e3653448eb808bc12f07b9fb9ae0"
server_key_iv: "8fac24571d7f34839adb66956753146f"
server_key_version: "encryptionkey@0"
I20260923 19:29:30.259771 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 19:29:30.263967 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:30.265796 3025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:29:30.265987 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "7bcf0a64443147acac9235e445a9f8e6"
format_stamp: "Formatted at 2026-09-23 19:29:30 on dist-test-slave-kxfb"
server_key: "2d64e3653448eb808bc12f07b9fb9ae0"
server_key_iv: "8fac24571d7f34839adb66956753146f"
server_key_version: "encryptionkey@0"
I20260923 19:29:30.266209 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:30.289233 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:30.294147 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:30.294492 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:30.295624 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:30.297132 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:30.297207 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:30.297336 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:30.297389 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:30.331166 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35583
I20260923 19:29:30.331218 3152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35583 every 8 connection(s)
I20260923 19:29:30.332649 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:30.333670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 3025
I20260923 19:29:30.333855 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:30.334467 388 external_mini_cluster.cc:1428] Setting key 074ec94f1e62c1aaa1eb052d93d1b0ca
Sep 23 19:29:30 dist-test-slave-kxfb krb5kdc[2785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:29:30.354909 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:30.335684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40501 (local address 127.0.97.62:34559)
0923 19:29:30.335883 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:30.335889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:30.337431 (+ 1542us) server_negotiation.cc:407] Connection header received
0923 19:29:30.339403 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:30.339408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:30.339487 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:30.339571 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:30.341162 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:30.342004 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:30.343716 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:30.344289 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:30.344410 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:30.348685 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:30.348691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:30.348694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:30.348735 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:30.351428 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:30.352321 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:30.352325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:30.352328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:30.352462 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:30.352952 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:30.352956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:30.352958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:30.353271 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:30.353405 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:30.354261 (+ 856us) server_negotiation.cc:299] Negotiation successful
0923 19:29:30.354547 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 19:29:30.356153 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:30.336358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34559 (local address 127.0.97.2:40501)
0923 19:29:30.337259 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:30.337305 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:30.339139 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:30.339824 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:30.339831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:30.340388 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:30.340994 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:30.341011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:30.342139 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:30.342143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:30.343481 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:30.343496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:30.344533 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:30.344540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:30.344809 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:30.345551 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:30.345584 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:30.348390 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:30.351643 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:30.351655 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:30.351676 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:30.352187 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:30.352628 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:30.352632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:30.352635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:30.352833 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:30.353387 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:30.353396 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:30.353868 (+ 472us) client_negotiation.cc:769] Sending connection context
0923 19:29:30.354149 (+ 281us) client_negotiation.cc:240] Negotiation successful
0923 19:29:30.354828 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":357,"threads_started":1}
I20260923 19:29:30.357903 3153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34559
I20260923 19:29:30.358345 3153 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:30.359292 3153 heartbeater.cc:511] Master 127.0.97.62:34559 requested a full tablet report, sending...
I20260923 19:29:30.361013 2830 ts_manager.cc:194] Registered new tserver with Master: 7bcf0a64443147acac9235e445a9f8e6 (127.0.97.2:35583)
I20260923 19:29:30.361920 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:29:30.377785 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34559
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:30.694084 3158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:30.694433 3158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:30.694581 3158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:30.701766 3158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:30.701902 3158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:30.702047 3158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:30.711865 3158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:30.713138 3158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:30.714385 3158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:30.727787 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:30.727960 3166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:30.727974 3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:30.728981 3158 server_base.cc:1034] running on GCE node
I20260923 19:29:30.729586 3158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:30.730602 3158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:30.731805 3158 hybrid_clock.cc:648] HybridClock initialized: now 1790191770731746 us; error 63 us; skew 500 ppm
I20260923 19:29:30.732120 3158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:30 dist-test-slave-kxfb krb5kdc[2785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:30.735327 3158 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:29:30.736861 3158 webserver.cc:460] Webserver started at http://127.0.97.3:45723/ using document root <none> and password file <none>
I20260923 19:29:30.737560 3158 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:30.737673 3158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:30.737962 3158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:30.740850 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "6b24d4dc0d354f18b7454dea6dab4579"
format_stamp: "Formatted at 2026-09-23 19:29:30 on dist-test-slave-kxfb"
server_key: "b840277914553dc6c1f6f4c11248b001"
server_key_iv: "fff4927b035a186dc4b18350585fb174"
server_key_version: "encryptionkey@0"
I20260923 19:29:30.741607 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6b24d4dc0d354f18b7454dea6dab4579"
format_stamp: "Formatted at 2026-09-23 19:29:30 on dist-test-slave-kxfb"
server_key: "b840277914553dc6c1f6f4c11248b001"
server_key_iv: "fff4927b035a186dc4b18350585fb174"
server_key_version: "encryptionkey@0"
I20260923 19:29:30.747391 3158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 19:29:30.751382 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:30.753059 3158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:29:30.753252 3158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "6b24d4dc0d354f18b7454dea6dab4579"
format_stamp: "Formatted at 2026-09-23 19:29:30 on dist-test-slave-kxfb"
server_key: "b840277914553dc6c1f6f4c11248b001"
server_key_iv: "fff4927b035a186dc4b18350585fb174"
server_key_version: "encryptionkey@0"
I20260923 19:29:30.753469 3158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:30.776503 3158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:30.780391 3158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:30.780624 3158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:30.781566 3158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:30.782917 3158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:30.782995 3158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:30.783110 3158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:30.783162 3158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:30.815637 3158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44781
I20260923 19:29:30.815733 3285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44781 every 8 connection(s)
I20260923 19:29:30.816882 3158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:30.817657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 3158
I20260923 19:29:30.817849 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:30.818454 388 external_mini_cluster.cc:1428] Setting key 926a0d533e7f17ecebdcdeeb38629a2b
Sep 23 19:29:30 dist-test-slave-kxfb krb5kdc[2785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:30.840394 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:30.819830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51241 (local address 127.0.97.62:34559)
0923 19:29:30.820041 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:30.820048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:30.824557 (+ 4509us) server_negotiation.cc:407] Connection header received
0923 19:29:30.826347 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:30.826353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:30.826447 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:30.826550 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:30.828157 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:30.829004 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:30.830714 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:30.831131 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:30.831234 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:30.835104 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:30.835109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:30.835112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:30.835157 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:30.837709 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:30.838390 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:30.838394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:30.838397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:30.838534 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:30.838885 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:30.838888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:30.838890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:30.839174 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:30.839289 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:30.839929 (+ 640us) server_negotiation.cc:299] Negotiation successful
0923 19:29:30.840122 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 19:29:30.841333 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:30.820663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34559 (local address 127.0.97.3:51241)
0923 19:29:30.824393 (+ 3730us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:30.824437 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:30.826033 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:30.826852 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:30.826861 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:30.827399 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:30.827979 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:30.827998 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:30.829204 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:30.829209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:30.830546 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:30.830560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:30.831371 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:30.831376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:30.831577 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:30.832259 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:30.832280 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:30.834913 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:30.837840 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:30.837847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:30.837860 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:30.838264 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:30.838628 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:30.838631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:30.838633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:30.838797 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:30.839274 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:30.839279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:30.839580 (+ 301us) client_negotiation.cc:769] Sending connection context
0923 19:29:30.839827 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 19:29:30.840209 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3455,"thread_start_us":3202,"threads_started":1}
I20260923 19:29:30.843004 3286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34559
I20260923 19:29:30.843357 3286 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:30.844085 3286 heartbeater.cc:511] Master 127.0.97.62:34559 requested a full tablet report, sending...
I20260923 19:29:30.845798 2830 ts_manager.cc:194] Registered new tserver with Master: 6b24d4dc0d354f18b7454dea6dab4579 (127.0.97.3:44781)
I20260923 19:29:30.846930 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51241
I20260923 19:29:30.847514 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:29:30.854611 3019 heartbeater.cc:503] Master 127.0.97.62:34559 was elected leader, sending a full tablet report...
W20260923 19:29:30.881263 3021 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:29:30.878562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.0.97.62:34559)
0923 19:29:30.878890 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:30.878899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:30.879306 (+ 407us) server_negotiation.cc:407] Connection header received
0923 19:29:30.879483 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:30.879488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:30.879568 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:30.879674 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:30.880967 (+ 1293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42656: BlockingRecv error: recv got EOF from 127.0.0.1:42656 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 19:29:30.884467 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2891
W20260923 19:29:30.887563 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2891: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:44399: server requires authentication, but client does not have Kerberos credentials available
I20260923 19:29:30.887748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 2891
I20260923 19:29:30.902233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3025
W20260923 19:29:30.905242 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3025: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:35583: server requires authentication, but client does not have Kerberos credentials available
W20260923 19:29:30.905419 3298 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:29:30.903391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.0.97.2:35583)
0923 19:29:30.903923 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:30.903930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:30.903961 (+ 31us) server_negotiation.cc:407] Connection header received
0923 19:29:30.904030 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:30.904037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:30.904266 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:30.904403 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:30.905114 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55446: BlockingRecv error: recv got EOF from 127.0.0.1:55446 (error 108)
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":259,"threads_started":1}
I20260923 19:29:30.905447 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 3025
I20260923 19:29:30.918524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3158
W20260923 19:29:30.921804 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3158: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:44781: server requires authentication, but client does not have Kerberos credentials available
I20260923 19:29:30.922015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 3158
I20260923 19:29:30.936076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 2798
I20260923 19:29:30.937294 2860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:31.049973 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 2798
2026-09-23T19:29:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3474 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:31 dist-test-slave-kxfb krb5kdc[3303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:31 dist-test-slave-kxfb krb5kdc[3303](info): set up 2 sockets
Sep 23 19:29:31 dist-test-slave-kxfb krb5kdc[3303](info): commencing operation
krb5kdc: starting...
W20260923 19:29:32.062899 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:32 dist-test-slave-kxfb krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:32.264098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46555
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:32.627491 3316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:32.627842 3316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:32.627982 3316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:32.635627 3316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:32.635752 3316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:32.635829 3316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:32.635913 3316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:32.635982 3316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:32.646756 3316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:32.648420 3316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:32.649945 3316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:32.662811 3324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:32.662973 3321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:32.662786 3322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:32.663612 3316 server_base.cc:1034] running on GCE node
I20260923 19:29:32.664623 3316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:32.666424 3316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:32.667623 3316 hybrid_clock.cc:648] HybridClock initialized: now 1790191772667557 us; error 58 us; skew 500 ppm
I20260923 19:29:32.668090 3316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:32 dist-test-slave-kxfb krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:32.676419 3316 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:32.678232 3316 webserver.cc:460] Webserver started at http://127.0.97.62:33059/ using document root <none> and password file <none>
I20260923 19:29:32.679109 3316 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:32.679221 3316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:32.679512 3316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:32.682978 3316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "af2fe67179564a7eaf626f1acc14a4d2"
format_stamp: "Formatted at 2026-09-23 19:29:32 on dist-test-slave-kxfb"
server_key: "e59bdafa09db1ff34c061694596c026e"
server_key_iv: "5622d65ded826363f68f81683eed0d9a"
server_key_version: "encryptionkey@0"
I20260923 19:29:32.683764 3316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "af2fe67179564a7eaf626f1acc14a4d2"
format_stamp: "Formatted at 2026-09-23 19:29:32 on dist-test-slave-kxfb"
server_key: "e59bdafa09db1ff34c061694596c026e"
server_key_iv: "5622d65ded826363f68f81683eed0d9a"
server_key_version: "encryptionkey@0"
I20260923 19:29:32.689541 3316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 19:29:32.693640 3331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:32.695254 3316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:32.695426 3316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "af2fe67179564a7eaf626f1acc14a4d2"
format_stamp: "Formatted at 2026-09-23 19:29:32 on dist-test-slave-kxfb"
server_key: "e59bdafa09db1ff34c061694596c026e"
server_key_iv: "5622d65ded826363f68f81683eed0d9a"
server_key_version: "encryptionkey@0"
I20260923 19:29:32.695645 3316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:32.713399 3316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:32.717891 3316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:32.718185 3316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:32.737152 3316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46555
I20260923 19:29:32.737185 3382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46555 every 8 connection(s)
I20260923 19:29:32.738775 3316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:32.743101 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:32.746003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 3316
I20260923 19:29:32.746236 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:32.747052 388 external_mini_cluster.cc:1428] Setting key cfb1f0d023f135d9662c3cbe73462844
I20260923 19:29:32.756484 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2: Bootstrap starting.
I20260923 19:29:32.760690 3383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:32.762105 3383 log.cc:824] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2: Log is configured to *not* fsync() on all Append() calls
Sep 23 19:29:32 dist-test-slave-kxfb krb5kdc[3303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:32.765365 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2: No bootstrap required, opened a new log
I20260923 19:29:32.772074 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:32.749209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.0.97.62:46555)
0923 19:29:32.750110 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:32.750123 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:32.750179 (+ 56us) server_negotiation.cc:407] Connection header received
0923 19:29:32.752129 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:32.752155 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:32.752654 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:32.753062 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:32.754944 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:32.756406 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:32.758120 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:32.758926 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:32.759072 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:32.764047 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:32.764056 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:32.764074 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:32.764132 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:32.767495 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:32.768239 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:32.768246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:32.768253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:32.768451 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:32.768852 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:32.768856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:32.768858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:32.769501 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:32.769847 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:32.770343 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 19:29:32.770817 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":412,"threads_started":1}
I20260923 19:29:32.773715 3383 raft_consensus.cc:348] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2fe67179564a7eaf626f1acc14a4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46555 } }
I20260923 19:29:32.774207 3383 raft_consensus.cc:374] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:32.774344 3383 raft_consensus.cc:729] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2fe67179564a7eaf626f1acc14a4d2, State: Initialized, Role: FOLLOWER
I20260923 19:29:32.775022 3383 consensus_queue.cc:260] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2fe67179564a7eaf626f1acc14a4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46555 } }
I20260923 19:29:32.775249 3383 raft_consensus.cc:388] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:32.775377 3383 raft_consensus.cc:482] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:32.775543 3383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:32.776388 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:29:32.777931 3383 raft_consensus.cc:504] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2fe67179564a7eaf626f1acc14a4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46555 } }
I20260923 19:29:32.778514 3383 leader_election.cc:302] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af2fe67179564a7eaf626f1acc14a4d2; no voters:
I20260923 19:29:32.779141 3383 leader_election.cc:288] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:32.779251 3388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:32.780359 3388 raft_consensus.cc:686] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [term 1 LEADER]: Becoming Leader. State: Replica: af2fe67179564a7eaf626f1acc14a4d2, State: Running, Role: LEADER
I20260923 19:29:32.780936 3388 consensus_queue.cc:237] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2fe67179564a7eaf626f1acc14a4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46555 } }
I20260923 19:29:32.781621 3383 sys_catalog.cc:564] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:32.783926 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af2fe67179564a7eaf626f1acc14a4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2fe67179564a7eaf626f1acc14a4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46555 } } }
I20260923 19:29:32.783891 3390 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af2fe67179564a7eaf626f1acc14a4d2. Latest consensus state: current_term: 1 leader_uuid: "af2fe67179564a7eaf626f1acc14a4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2fe67179564a7eaf626f1acc14a4d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46555 } } }
I20260923 19:29:32.784380 3390 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:32.784390 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:32.793396 3395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:32.799923 3395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:32.819094 3395 catalog_manager.cc:1347] Generated new cluster ID: 9085c1ea53294495ad22cee08bf9cca1
I20260923 19:29:32.819239 3395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:32.837687 3395 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:32.839345 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:32.828897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.0.97.62:46555)
0923 19:29:32.829087 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:32.829093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:32.829836 (+ 743us) server_negotiation.cc:407] Connection header received
0923 19:29:32.830044 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:32.830050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:32.830145 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:32.830264 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:32.831578 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:32.832795 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:32.834694 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:32.835382 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:32.835526 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:32.837243 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:32.837250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:32.837255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:32.838367 (+ 1112us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:32.838476 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:32.838517 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:32.838627 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:32.838845 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 19:29:32.838960 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 19:29:32.839540 3395 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:32.853790 3395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af2fe67179564a7eaf626f1acc14a4d2: Generated new TSK 0
I20260923 19:29:32.855151 3395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:29:32.867118 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3316
I20260923 19:29:32.868373 3378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:32.965409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 3316
2026-09-23T19:29:32Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1909 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:33 dist-test-slave-kxfb krb5kdc[3417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:33 dist-test-slave-kxfb krb5kdc[3417](info): set up 2 sockets
Sep 23 19:29:33 dist-test-slave-kxfb krb5kdc[3417](info): commencing operation
krb5kdc: starting...
W20260923 19:29:33.966917 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:34 dist-test-slave-kxfb krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:34.174798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:34.535778 3430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:34.536139 3430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:34.536271 3430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:34.544107 3430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:34.544267 3430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:34.544333 3430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:34.544420 3430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:34.544482 3430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:34.555330 3430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:34.557049 3430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:34.558630 3430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:34.571667 3435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:34.571636 3436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:34.571705 3438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:34.572564 3430 server_base.cc:1034] running on GCE node
I20260923 19:29:34.573477 3430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:34.575243 3430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:34.576514 3430 hybrid_clock.cc:648] HybridClock initialized: now 1790191774576410 us; error 50 us; skew 500 ppm
I20260923 19:29:34.576900 3430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:34 dist-test-slave-kxfb krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:34.580379 3430 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:34.582191 3430 webserver.cc:460] Webserver started at http://127.0.97.62:39319/ using document root <none> and password file <none>
I20260923 19:29:34.583055 3430 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:34.583190 3430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:34.583529 3430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:34.587247 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "c976f0e94cb4421d8e9db77c2b6d48ad"
format_stamp: "Formatted at 2026-09-23 19:29:34 on dist-test-slave-kxfb"
server_key: "bf131759e95340e34d808753c42a7f40"
server_key_iv: "e3cfbb25a9677a07ef7a6deb8675cf42"
server_key_version: "encryptionkey@0"
I20260923 19:29:34.588063 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "c976f0e94cb4421d8e9db77c2b6d48ad"
format_stamp: "Formatted at 2026-09-23 19:29:34 on dist-test-slave-kxfb"
server_key: "bf131759e95340e34d808753c42a7f40"
server_key_iv: "e3cfbb25a9677a07ef7a6deb8675cf42"
server_key_version: "encryptionkey@0"
I20260923 19:29:34.594038 3430 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260923 19:29:34.597996 3445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:34.599563 3430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:34.599741 3430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "c976f0e94cb4421d8e9db77c2b6d48ad"
format_stamp: "Formatted at 2026-09-23 19:29:34 on dist-test-slave-kxfb"
server_key: "bf131759e95340e34d808753c42a7f40"
server_key_iv: "e3cfbb25a9677a07ef7a6deb8675cf42"
server_key_version: "encryptionkey@0"
I20260923 19:29:34.599957 3430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:34.618005 3430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:34.622972 3430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:34.623332 3430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:34.642935 3430 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33219
I20260923 19:29:34.642958 3496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33219 every 8 connection(s)
I20260923 19:29:34.644608 3430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:34.645987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 3430
I20260923 19:29:34.646224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:34.647127 388 external_mini_cluster.cc:1428] Setting key 95393d73c3796ac967aaad79ee00556a
I20260923 19:29:34.650012 3497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 19:29:34 dist-test-slave-kxfb krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:34.664574 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad: Bootstrap starting.
I20260923 19:29:34.669445 3497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:34.670933 3497 log.cc:824] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:34.674856 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad: No bootstrap required, opened a new log
I20260923 19:29:34.674878 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:34.649525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.0.97.62:33219)
0923 19:29:34.650439 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:34.650454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:34.650501 (+ 47us) server_negotiation.cc:407] Connection header received
0923 19:29:34.652233 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:34.652258 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:34.652706 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:34.653122 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:34.654974 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:34.656484 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:34.658231 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:34.659045 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:34.659190 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:34.664627 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:34.664641 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:34.664668 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:34.664751 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:34.669677 (+ 4926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:34.670530 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:34.670541 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:34.670553 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:34.670786 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:34.671195 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:34.671199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:34.671202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:34.671974 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:34.672231 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:34.672802 (+ 571us) server_negotiation.cc:299] Negotiation successful
0923 19:29:34.673327 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":380,"threads_started":1}
I20260923 19:29:34.683036 3497 raft_consensus.cc:348] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c976f0e94cb4421d8e9db77c2b6d48ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } }
I20260923 19:29:34.683471 3497 raft_consensus.cc:374] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:34.683553 3497 raft_consensus.cc:729] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c976f0e94cb4421d8e9db77c2b6d48ad, State: Initialized, Role: FOLLOWER
I20260923 19:29:34.684151 3497 consensus_queue.cc:260] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c976f0e94cb4421d8e9db77c2b6d48ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } }
I20260923 19:29:34.684378 3497 raft_consensus.cc:388] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:34.684522 3497 raft_consensus.cc:482] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:34.684689 3497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:34.687085 3497 raft_consensus.cc:504] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c976f0e94cb4421d8e9db77c2b6d48ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } }
I20260923 19:29:34.687631 3497 leader_election.cc:302] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c976f0e94cb4421d8e9db77c2b6d48ad; no voters:
I20260923 19:29:34.688251 3497 leader_election.cc:288] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:34.688382 3502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:34.689042 3502 raft_consensus.cc:686] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [term 1 LEADER]: Becoming Leader. State: Replica: c976f0e94cb4421d8e9db77c2b6d48ad, State: Running, Role: LEADER
I20260923 19:29:34.689633 3502 consensus_queue.cc:237] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c976f0e94cb4421d8e9db77c2b6d48ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } }
I20260923 19:29:34.690460 3497 sys_catalog.cc:564] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:34.692704 3503 sys_catalog.cc:455] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c976f0e94cb4421d8e9db77c2b6d48ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c976f0e94cb4421d8e9db77c2b6d48ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } }
I20260923 19:29:34.692623 3504 sys_catalog.cc:455] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c976f0e94cb4421d8e9db77c2b6d48ad. Latest consensus state: current_term: 1 leader_uuid: "c976f0e94cb4421d8e9db77c2b6d48ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c976f0e94cb4421d8e9db77c2b6d48ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33219 } } }
I20260923 19:29:34.693110 3503 sys_catalog.cc:458] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:34.693214 3504 sys_catalog.cc:458] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:34.695905 3509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:34.702092 3509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:34.712097 3509 catalog_manager.cc:1347] Generated new cluster ID: 4b293256ef584452a79695ea07233dd6
I20260923 19:29:34.712265 3509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:34.724432 3509 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:34.726017 3509 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:34.733250 3509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c976f0e94cb4421d8e9db77c2b6d48ad: Generated new TSK 0
I20260923 19:29:34.734498 3509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:29:34.796180 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33219
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:35.184592 3523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:35.184940 3523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:35.185071 3523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:35.192866 3523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:35.192998 3523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:35.193150 3523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:35.204197 3523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:35.205933 3523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:35.207530 3523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:35.221745 3531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:35.221800 3529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:35.221840 3528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:35.222738 3523 server_base.cc:1034] running on GCE node
I20260923 19:29:35.223443 3523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:35.224566 3523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:35.225765 3523 hybrid_clock.cc:648] HybridClock initialized: now 1790191775225716 us; error 46 us; skew 500 ppm
I20260923 19:29:35.226090 3523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:35 dist-test-slave-kxfb krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:35.229669 3523 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:29:35.231326 3523 webserver.cc:460] Webserver started at http://127.0.97.1:35853/ using document root <none> and password file <none>
I20260923 19:29:35.232154 3523 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:35.232268 3523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:35.232606 3523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:35.236030 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "e59f5915e1204aadb43f39c53ee8a7dc"
format_stamp: "Formatted at 2026-09-23 19:29:35 on dist-test-slave-kxfb"
server_key: "a6c06cd9e3a8636720bc7f745559e8e2"
server_key_iv: "76478b3e9c071b6400e1e0fc42715aa2"
server_key_version: "encryptionkey@0"
I20260923 19:29:35.236858 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e59f5915e1204aadb43f39c53ee8a7dc"
format_stamp: "Formatted at 2026-09-23 19:29:35 on dist-test-slave-kxfb"
server_key: "a6c06cd9e3a8636720bc7f745559e8e2"
server_key_iv: "76478b3e9c071b6400e1e0fc42715aa2"
server_key_version: "encryptionkey@0"
I20260923 19:29:35.243516 3523 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 19:29:35.247566 3538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:35.249248 3523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:29:35.249477 3523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "e59f5915e1204aadb43f39c53ee8a7dc"
format_stamp: "Formatted at 2026-09-23 19:29:35 on dist-test-slave-kxfb"
server_key: "a6c06cd9e3a8636720bc7f745559e8e2"
server_key_iv: "76478b3e9c071b6400e1e0fc42715aa2"
server_key_version: "encryptionkey@0"
I20260923 19:29:35.249693 3523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:35.262377 3523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:35.266377 3523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:35.266649 3523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:35.267647 3523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:35.269039 3523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:35.269116 3523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:35.269222 3523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:35.269284 3523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:35.303416 3523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43347
I20260923 19:29:35.303499 3650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43347 every 8 connection(s)
I20260923 19:29:35.304817 3523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:35.310758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 3523
I20260923 19:29:35.310979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:35.311640 388 external_mini_cluster.cc:1428] Setting key 8cea46f3c982494d0a96555e7f73c2c8
Sep 23 19:29:35 dist-test-slave-kxfb krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:35.327956 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:35.307778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52083 (local address 127.0.97.62:33219)
0923 19:29:35.308207 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:35.308211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:35.309324 (+ 1113us) server_negotiation.cc:407] Connection header received
0923 19:29:35.311326 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:35.311332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:35.311430 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:35.311543 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:35.313375 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:35.314629 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:35.316212 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:35.316833 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:35.316987 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:35.321279 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:35.321287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:35.321293 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:35.321338 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:35.324531 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:35.325448 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:35.325454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:35.325457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:35.325606 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:35.326027 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:35.326031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:35.326034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:35.326403 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:35.326545 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:35.327338 (+ 793us) server_negotiation.cc:299] Negotiation successful
0923 19:29:35.327580 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
I20260923 19:29:35.329234 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:35.308314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.97.1:52083)
0923 19:29:35.309120 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:35.309165 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:35.310994 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:35.311839 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:35.311851 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:35.312427 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:35.313141 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:35.313166 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:35.314806 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:35.314810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:35.316027 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:35.316040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:35.317068 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:35.317073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:35.317275 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:35.318020 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:35.318042 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:35.320967 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:35.324731 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:35.324742 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:35.324760 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:35.325324 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:35.325716 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:35.325719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:35.325722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:35.325917 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:35.326528 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:35.326536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:35.326932 (+ 396us) client_negotiation.cc:769] Sending connection context
0923 19:29:35.327206 (+ 274us) client_negotiation.cc:240] Negotiation successful
0923 19:29:35.327741 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":299,"threads_started":1}
I20260923 19:29:35.331169 3651 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33219
I20260923 19:29:35.331511 3651 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 19:29:35.332372 3651 heartbeater.cc:511] Master 127.0.97.62:33219 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 19:29:35.335788 3462 ts_manager.cc:194] Registered new tserver with Master: e59f5915e1204aadb43f39c53ee8a7dc (127.0.97.1:43347)
I20260923 19:29:35.337975 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:29:35.361521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33219
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:35.698804 3657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:35.699082 3657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:35.699186 3657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:35.706512 3657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:35.706629 3657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:35.706773 3657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:35.716387 3657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:35.717713 3657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:35.719033 3657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:35.732230 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:35.732381 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:35.732448 3663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:35.733647 3657 server_base.cc:1034] running on GCE node
I20260923 19:29:35.734277 3657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:35.735280 3657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:35.736521 3657 hybrid_clock.cc:648] HybridClock initialized: now 1790191775736437 us; error 66 us; skew 500 ppm
I20260923 19:29:35.736837 3657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:35 dist-test-slave-kxfb krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:35.740321 3657 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:29:35.742105 3657 webserver.cc:460] Webserver started at http://127.0.97.2:32831/ using document root <none> and password file <none>
I20260923 19:29:35.742959 3657 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:35.743076 3657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:35.743366 3657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:35.746740 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "ffc9736fd356426786495ba6c7af75fc"
format_stamp: "Formatted at 2026-09-23 19:29:35 on dist-test-slave-kxfb"
server_key: "537abd72590d254d141c5ba3f4e8b977"
server_key_iv: "d5b3b1f09ec6d4ce6a8ddc1b24960443"
server_key_version: "encryptionkey@0"
I20260923 19:29:35.747550 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ffc9736fd356426786495ba6c7af75fc"
format_stamp: "Formatted at 2026-09-23 19:29:35 on dist-test-slave-kxfb"
server_key: "537abd72590d254d141c5ba3f4e8b977"
server_key_iv: "d5b3b1f09ec6d4ce6a8ddc1b24960443"
server_key_version: "encryptionkey@0"
I20260923 19:29:35.753417 3657 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 19:29:35.757746 3672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:35.759500 3657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 19:29:35.759665 3657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "ffc9736fd356426786495ba6c7af75fc"
format_stamp: "Formatted at 2026-09-23 19:29:35 on dist-test-slave-kxfb"
server_key: "537abd72590d254d141c5ba3f4e8b977"
server_key_iv: "d5b3b1f09ec6d4ce6a8ddc1b24960443"
server_key_version: "encryptionkey@0"
I20260923 19:29:35.759871 3657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:35.806615 3657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:35.811448 3657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:35.811780 3657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:35.812917 3657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:35.814497 3657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:35.814577 3657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:35.814682 3657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:35.814735 3657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:35.849463 3657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43491
I20260923 19:29:35.849521 3784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43491 every 8 connection(s)
I20260923 19:29:35.850937 3657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:35.852897 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 3657
I20260923 19:29:35.853097 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:35.853581 388 external_mini_cluster.cc:1428] Setting key 7950975873270f673e367189dec2935d
Sep 23 19:29:35 dist-test-slave-kxfb krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:35.870918 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:35.853761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57059 (local address 127.0.97.62:33219)
0923 19:29:35.854371 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:35.854378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:35.855365 (+ 987us) server_negotiation.cc:407] Connection header received
0923 19:29:35.857191 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:35.857197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:35.857290 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:35.857383 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:35.858879 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:35.859607 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:35.861074 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:35.861505 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:35.861608 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:35.865450 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:35.865457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:35.865460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:35.865493 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:35.867906 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:35.868638 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:35.868644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:35.868646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:35.868769 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:35.869178 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:35.869181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:35.869183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:35.869458 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:35.869587 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:35.870317 (+ 730us) server_negotiation.cc:299] Negotiation successful
0923 19:29:35.870545 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":317,"threads_started":1}
I20260923 19:29:35.871938 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:35.854380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.97.2:57059)
0923 19:29:35.855182 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:35.855241 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:35.856917 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:35.857611 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:35.857619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:35.858124 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:35.858715 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:35.858732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:35.859739 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:35.859743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:35.860946 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:35.860958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:35.861705 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:35.861712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:35.861934 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:35.862713 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:35.862736 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:35.865207 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:35.868059 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:35.868069 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:35.868085 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:35.868515 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:35.868872 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:35.868878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:35.868882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:35.869064 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:35.869580 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:35.869588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:35.869928 (+ 340us) client_negotiation.cc:769] Sending connection context
0923 19:29:35.870159 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 19:29:35.870729 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":319,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:29:35.873750 3785 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33219
I20260923 19:29:35.874190 3785 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 19:29:35.875247 3785 heartbeater.cc:511] Master 127.0.97.62:33219 requested a full tablet report, sending...
I20260923 19:29:35.876919 3462 ts_manager.cc:194] Registered new tserver with Master: ffc9736fd356426786495ba6c7af75fc (127.0.97.2:43491)
I20260923 19:29:35.877822 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:29:35.899186 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33219
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:36.267094 3791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:36.267408 3791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:36.267535 3791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:36.275686 3791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:36.275807 3791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:36.275951 3791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:36.285934 3791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:36.287351 3791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:36.288671 3791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:36.302556 3797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:36.302598 3796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:36.302846 3799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:36.303656 3791 server_base.cc:1034] running on GCE node
I20260923 19:29:36.304469 3791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:36.305613 3791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:36.306849 3791 hybrid_clock.cc:648] HybridClock initialized: now 1790191776306778 us; error 66 us; skew 500 ppm
I20260923 19:29:36.307237 3791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:36.310523 3791 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:29:36.311941 3791 webserver.cc:460] Webserver started at http://127.0.97.3:44189/ using document root <none> and password file <none>
I20260923 19:29:36.312613 3791 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:36.312718 3791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:36.312959 3791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:36.315836 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "376c28b672dc4005aee0ab6dc8d2cb76"
format_stamp: "Formatted at 2026-09-23 19:29:36 on dist-test-slave-kxfb"
server_key: "04bc881234cfebbd04b47365919be4b7"
server_key_iv: "ccb3ef8da967c4226ec37f0637d674cf"
server_key_version: "encryptionkey@0"
I20260923 19:29:36.316550 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "376c28b672dc4005aee0ab6dc8d2cb76"
format_stamp: "Formatted at 2026-09-23 19:29:36 on dist-test-slave-kxfb"
server_key: "04bc881234cfebbd04b47365919be4b7"
server_key_iv: "ccb3ef8da967c4226ec37f0637d674cf"
server_key_version: "encryptionkey@0"
I20260923 19:29:36.321887 3791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 19:29:36.325600 3806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:36.327280 3791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 19:29:36.327445 3791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "376c28b672dc4005aee0ab6dc8d2cb76"
format_stamp: "Formatted at 2026-09-23 19:29:36 on dist-test-slave-kxfb"
server_key: "04bc881234cfebbd04b47365919be4b7"
server_key_iv: "ccb3ef8da967c4226ec37f0637d674cf"
server_key_version: "encryptionkey@0"
I20260923 19:29:36.327678 3791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:36.342393 3651 heartbeater.cc:503] Master 127.0.97.62:33219 was elected leader, sending a full tablet report...
I20260923 19:29:36.362246 3791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:36.367007 3791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:36.367305 3791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:36.368405 3791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:36.369859 3791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:36.369933 3791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:36.370040 3791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:36.370105 3791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:36.403590 3791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42169
I20260923 19:29:36.403702 3918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42169 every 8 connection(s)
I20260923 19:29:36.404955 3791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:36.409310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 3791
I20260923 19:29:36.409546 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:36.410168 388 external_mini_cluster.cc:1428] Setting key 2e96a2381ee5c1972e9e594fbbb1ce9d
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:36.427042 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:36.407905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41889 (local address 127.0.97.62:33219)
0923 19:29:36.408434 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:36.408440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:36.409750 (+ 1310us) server_negotiation.cc:407] Connection header received
0923 19:29:36.411995 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:36.412000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:36.412098 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:36.412231 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:36.413958 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.414972 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.416569 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.417023 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.417129 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:36.420556 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:36.420563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:36.420567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:36.420607 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:36.423652 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.424508 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.424512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.424514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.424643 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.425147 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.425152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.425154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.425464 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:36.425595 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:36.426387 (+ 792us) server_negotiation.cc:299] Negotiation successful
0923 19:29:36.426626 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":262,"threads_started":1}
I20260923 19:29:36.428222 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:36.408489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33219 (local address 127.0.97.3:41889)
0923 19:29:36.409519 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:36.409600 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:36.411676 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:36.412514 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:36.412527 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:36.413032 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:36.413757 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:36.413778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.415139 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.415143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:36.416401 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:36.416414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.417260 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.417265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:36.417461 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:36.418218 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:36.418246 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:36.420295 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:36.423806 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:36.423815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:36.423832 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:36.424379 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:36.424784 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:36.424790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:36.424794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:36.425038 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:36.425590 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:36.425600 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:36.426002 (+ 402us) client_negotiation.cc:769] Sending connection context
0923 19:29:36.426277 (+ 275us) client_negotiation.cc:240] Negotiation successful
0923 19:29:36.426951 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":365,"threads_started":1}
I20260923 19:29:36.429711 3919 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33219
I20260923 19:29:36.430091 3919 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:36.430951 3919 heartbeater.cc:511] Master 127.0.97.62:33219 requested a full tablet report, sending...
I20260923 19:29:36.432438 3462 ts_manager.cc:194] Registered new tserver with Master: 376c28b672dc4005aee0ab6dc8d2cb76 (127.0.97.3:42169)
I20260923 19:29:36.433249 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41889
I20260923 19:29:36.437693 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191776, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191776, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:36.488541 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:36.474441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.0.97.62:33219)
0923 19:29:36.474726 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:36.474733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:36.475203 (+ 470us) server_negotiation.cc:407] Connection header received
0923 19:29:36.475393 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:36.475399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:36.475511 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:36.475643 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:36.477016 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.477838 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.479141 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.479567 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.479668 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:36.483522 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:36.483527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:36.483531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:36.483568 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:36.486102 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.486843 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.486847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.486850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.486977 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.487260 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.487263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.487265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.487607 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:36.487814 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:36.488011 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 19:29:36.488213 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260923 19:29:36.489223 3462 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44776
I20260923 19:29:36.490753 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3523
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191776, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:36.507771 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:36.491979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.0.97.1:43347)
0923 19:29:36.492585 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:36.492593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:36.492621 (+ 28us) server_negotiation.cc:407] Connection header received
0923 19:29:36.492681 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:36.492695 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:36.492900 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:36.493036 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:36.494211 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.495321 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.496986 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.497547 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.497649 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:36.502109 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:36.502121 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:36.502137 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:36.502178 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:36.505076 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.505848 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.505855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.505857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.505978 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.506344 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.506349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.506352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.506692 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:36.506920 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:36.507213 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 19:29:36.507407 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":322,"threads_started":1}
W20260923 19:29:36.509130 3646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55200
W20260923 19:29:36.510550 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3523: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:29:36.510747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 3523
I20260923 19:29:36.524060 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3657
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191776, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:29:36.541715 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:36.525102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.0.97.2:43491)
0923 19:29:36.525813 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:36.525822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:36.525854 (+ 32us) server_negotiation.cc:407] Connection header received
0923 19:29:36.525931 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:36.525940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:36.526231 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:36.526424 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:36.527643 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.528920 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.530266 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.530978 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.531163 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:36.535468 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:36.535481 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:36.535500 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:36.535539 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:36.538827 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.539516 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.539525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.539527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.539674 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.540063 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.540067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.540069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.540498 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:36.540755 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:36.541070 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 19:29:36.541303 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":376,"threads_started":1}
W20260923 19:29:36.543221 3780 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59758
W20260923 19:29:36.544521 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3657: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:29:36.544720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 3657
I20260923 19:29:36.557193 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3791
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191776, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:29:36.574668 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:36.558273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.0.97.3:42169)
0923 19:29:36.558987 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:36.558994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:36.559028 (+ 34us) server_negotiation.cc:407] Connection header received
0923 19:29:36.559099 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:36.559106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:36.559315 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:36.559480 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:36.560940 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.561982 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.563437 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:36.564025 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:36.564141 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:36.568797 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:36.568810 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:36.568826 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:36.568864 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:36.571619 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.572410 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.572418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.572420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.572554 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:36.572935 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:36.572938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:36.572940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:36.573298 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:36.573531 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:36.573993 (+ 462us) server_negotiation.cc:299] Negotiation successful
0923 19:29:36.574261 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":392,"threads_started":1}
W20260923 19:29:36.575906 3914 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57518
W20260923 19:29:36.577059 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3791: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:29:36.577239 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 3791
I20260923 19:29:36.589970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3430
I20260923 19:29:36.591416 3492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:36.697026 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 3430
2026-09-23T19:29:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3737 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3937](info): set up 2 sockets
Sep 23 19:29:36 dist-test-slave-kxfb krb5kdc[3937](info): commencing operation
krb5kdc: starting...
W20260923 19:29:37.698782 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:37 dist-test-slave-kxfb krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:37.902228 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33907
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:38.232322 3950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:38.232620 3950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:38.232719 3950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:38.240010 3950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:38.240126 3950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:38.240172 3950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:38.240272 3950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:38.240336 3950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:38.250380 3950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:38.251705 3950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:38.252976 3950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:38.265148 3955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:38.265239 3956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:38.265196 3958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:38.266240 3950 server_base.cc:1034] running on GCE node
I20260923 19:29:38.267042 3950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:38.268401 3950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:38.269611 3950 hybrid_clock.cc:648] HybridClock initialized: now 1790191778269568 us; error 58 us; skew 500 ppm
I20260923 19:29:38.269943 3950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:38 dist-test-slave-kxfb krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:38.277082 3950 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:38.279003 3950 webserver.cc:460] Webserver started at http://127.0.97.62:39297/ using document root <none> and password file <none>
I20260923 19:29:38.279919 3950 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:38.280040 3950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:38.280354 3950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:38.283905 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "871d646d4b224b04a540fcd2da54b860"
format_stamp: "Formatted at 2026-09-23 19:29:38 on dist-test-slave-kxfb"
server_key: "1e98b76de2c095fb09d0d7601979eaf7"
server_key_iv: "864a7975d5a6cc50fea95a7f6de1730c"
server_key_version: "encryptionkey@0"
I20260923 19:29:38.284789 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "871d646d4b224b04a540fcd2da54b860"
format_stamp: "Formatted at 2026-09-23 19:29:38 on dist-test-slave-kxfb"
server_key: "1e98b76de2c095fb09d0d7601979eaf7"
server_key_iv: "864a7975d5a6cc50fea95a7f6de1730c"
server_key_version: "encryptionkey@0"
I20260923 19:29:38.290994 3950 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 19:29:38.295101 3965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:38.296854 3950 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260923 19:29:38.297051 3950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "871d646d4b224b04a540fcd2da54b860"
format_stamp: "Formatted at 2026-09-23 19:29:38 on dist-test-slave-kxfb"
server_key: "1e98b76de2c095fb09d0d7601979eaf7"
server_key_iv: "864a7975d5a6cc50fea95a7f6de1730c"
server_key_version: "encryptionkey@0"
I20260923 19:29:38.297258 3950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:38.333621 3950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:38.337764 3950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:38.338009 3950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:38.357193 3950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33907
I20260923 19:29:38.357271 4016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33907 every 8 connection(s)
I20260923 19:29:38.358717 3950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:38.361140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 3950
I20260923 19:29:38.361317 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:38.361919 388 external_mini_cluster.cc:1428] Setting key 34b29d47c8eabfd123fafd4a3353c0dd
I20260923 19:29:38.363705 4017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 19:29:38 dist-test-slave-kxfb krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:38.378396 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860: Bootstrap starting.
I20260923 19:29:38.382385 4017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:38.383550 4017 log.cc:824] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:38.386361 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:38.364164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.0.97.62:33907)
0923 19:29:38.365130 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:38.365148 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:38.365213 (+ 65us) server_negotiation.cc:407] Connection header received
0923 19:29:38.367394 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:38.367425 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:38.368068 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:38.368649 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:38.370157 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:38.372261 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:38.373528 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:38.374608 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:38.374808 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:38.378731 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:38.378742 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:38.378763 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:38.378817 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:38.382076 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:38.382876 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:38.382883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:38.382890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:38.383057 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:38.383404 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:38.383408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:38.383410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:38.384021 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:38.384338 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:38.384819 (+ 481us) server_negotiation.cc:299] Negotiation successful
0923 19:29:38.385236 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":374,"threads_started":1}
I20260923 19:29:38.387174 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860: No bootstrap required, opened a new log
I20260923 19:29:38.395778 4017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d646d4b224b04a540fcd2da54b860" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33907 } }
I20260923 19:29:38.396325 4017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:38.396445 4017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871d646d4b224b04a540fcd2da54b860, State: Initialized, Role: FOLLOWER
I20260923 19:29:38.397151 4017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d646d4b224b04a540fcd2da54b860" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33907 } }
I20260923 19:29:38.397403 4017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:38.397554 4017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:38.397724 4017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:38.400143 4017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d646d4b224b04a540fcd2da54b860" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33907 } }
I20260923 19:29:38.400735 4017 leader_election.cc:302] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 871d646d4b224b04a540fcd2da54b860; no voters:
I20260923 19:29:38.401420 4017 leader_election.cc:288] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:38.401592 4022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:38.402693 4022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [term 1 LEADER]: Becoming Leader. State: Replica: 871d646d4b224b04a540fcd2da54b860, State: Running, Role: LEADER
I20260923 19:29:38.403324 4022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d646d4b224b04a540fcd2da54b860" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33907 } }
I20260923 19:29:38.403990 4017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:38.406265 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "871d646d4b224b04a540fcd2da54b860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d646d4b224b04a540fcd2da54b860" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33907 } } }
I20260923 19:29:38.406762 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:38.406306 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871d646d4b224b04a540fcd2da54b860. Latest consensus state: current_term: 1 leader_uuid: "871d646d4b224b04a540fcd2da54b860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d646d4b224b04a540fcd2da54b860" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33907 } } }
I20260923 19:29:38.407847 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:38.411437 4029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:38.417016 4029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:38.426440 4029 catalog_manager.cc:1347] Generated new cluster ID: 1a57b8e7d778434399da47c8bb6489b6
I20260923 19:29:38.426581 4029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:38.453296 4029 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:38.455394 4029 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:38.463228 4029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 871d646d4b224b04a540fcd2da54b860: Generated new TSK 0
I20260923 19:29:38.464349 4029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:29:38.524425 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33907
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:38.898571 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:38.898947 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:38.899099 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:38.907348 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:38.907505 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:38.907668 4043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:38.918642 4043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:38.920302 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:38.921875 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:38.937155 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:38.937264 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:38.937305 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:38.938319 4043 server_base.cc:1034] running on GCE node
I20260923 19:29:38.938994 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:38.940012 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:38.941248 4043 hybrid_clock.cc:648] HybridClock initialized: now 1790191778941187 us; error 56 us; skew 500 ppm
I20260923 19:29:38.941593 4043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:38 dist-test-slave-kxfb krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:38.944829 4043 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:29:38.946538 4043 webserver.cc:460] Webserver started at http://127.0.97.1:38367/ using document root <none> and password file <none>
I20260923 19:29:38.947268 4043 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:38.947371 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:38.947650 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:38.950731 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "a9bd101876994cb19ebe1ff484b77b12"
format_stamp: "Formatted at 2026-09-23 19:29:38 on dist-test-slave-kxfb"
server_key: "239455cffc40f9946c60a2e39ccddfa2"
server_key_iv: "8d19954c5fbec81047ffa9fbe5b57cb0"
server_key_version: "encryptionkey@0"
I20260923 19:29:38.951494 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a9bd101876994cb19ebe1ff484b77b12"
format_stamp: "Formatted at 2026-09-23 19:29:38 on dist-test-slave-kxfb"
server_key: "239455cffc40f9946c60a2e39ccddfa2"
server_key_iv: "8d19954c5fbec81047ffa9fbe5b57cb0"
server_key_version: "encryptionkey@0"
I20260923 19:29:38.957032 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 19:29:38.960825 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:38.962495 4043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:38.962682 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "a9bd101876994cb19ebe1ff484b77b12"
format_stamp: "Formatted at 2026-09-23 19:29:38 on dist-test-slave-kxfb"
server_key: "239455cffc40f9946c60a2e39ccddfa2"
server_key_iv: "8d19954c5fbec81047ffa9fbe5b57cb0"
server_key_version: "encryptionkey@0"
I20260923 19:29:38.962879 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:38.978713 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:38.983052 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:38.983340 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:38.984352 4043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:38.985738 4043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:38.985816 4043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:38.985940 4043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:38.985992 4043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:39.018796 4043 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34171
I20260923 19:29:39.018901 4170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34171 every 8 connection(s)
I20260923 19:29:39.020093 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:39.027248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 4043
I20260923 19:29:39.027489 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:39.028177 388 external_mini_cluster.cc:1428] Setting key 09be7fe5d66ad3be464a88c9b6e7f588
Sep 23 19:29:39 dist-test-slave-kxfb krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:39.040359 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:39.023085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51859 (local address 127.0.97.62:33907)
0923 19:29:39.023602 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:39.023607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:39.024757 (+ 1150us) server_negotiation.cc:407] Connection header received
0923 19:29:39.026613 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:39.026618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:39.026702 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:39.026795 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:39.028433 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:39.029248 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:39.030933 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:39.031343 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:39.031453 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:39.034856 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:39.034861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:39.034864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:39.034895 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:39.037510 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:39.038171 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:39.038175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:39.038177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:39.038323 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:39.038721 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:39.038726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:39.038729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:39.038969 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:39.039105 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:39.039773 (+ 668us) server_negotiation.cc:299] Negotiation successful
0923 19:29:39.039997 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":272,"threads_started":1}
I20260923 19:29:39.041317 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:39.023703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33907 (local address 127.0.97.1:51859)
0923 19:29:39.024558 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:39.024602 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:39.026332 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:39.027043 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:39.027053 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:39.027651 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:39.028261 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:39.028287 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:39.029423 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:39.029427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:39.030802 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:39.030815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:39.031559 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:39.031566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:39.031801 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:39.032533 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:39.032553 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:39.034646 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:39.037631 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:39.037640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:39.037657 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:39.038051 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:39.038426 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:39.038431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:39.038435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:39.038608 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:39.039090 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:39.039098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:39.039409 (+ 311us) client_negotiation.cc:769] Sending connection context
0923 19:29:39.039641 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 19:29:39.040154 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":350,"threads_started":1}
I20260923 19:29:39.043159 4171 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33907
I20260923 19:29:39.043673 4171 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:39.044888 4171 heartbeater.cc:511] Master 127.0.97.62:33907 requested a full tablet report, sending...
I20260923 19:29:39.047978 3982 ts_manager.cc:194] Registered new tserver with Master: a9bd101876994cb19ebe1ff484b77b12 (127.0.97.1:34171)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 19:29:39.049757 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:29:39.075739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33907
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:39.436081 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:39.436370 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:39.436480 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:39.443610 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:39.443725 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:39.443866 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:39.453354 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:39.454725 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:39.456036 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:39.469791 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:39.470072 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:39.470209 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:39.470856 4177 server_base.cc:1034] running on GCE node
I20260923 19:29:39.471573 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:39.472681 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:39.473889 4177 hybrid_clock.cc:648] HybridClock initialized: now 1790191779473831 us; error 54 us; skew 500 ppm
I20260923 19:29:39.474236 4177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:39 dist-test-slave-kxfb krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:39.477627 4177 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:29:39.479287 4177 webserver.cc:460] Webserver started at http://127.0.97.2:40897/ using document root <none> and password file <none>
I20260923 19:29:39.480058 4177 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:39.480195 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:39.480533 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:39.483912 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "130ea167f9b742ba94e63909a158e8b7"
format_stamp: "Formatted at 2026-09-23 19:29:39 on dist-test-slave-kxfb"
server_key: "08985e067fc9da42b0bddd0385c4b0c0"
server_key_iv: "06f3b5e5daa472bdd7217f70c68b313a"
server_key_version: "encryptionkey@0"
I20260923 19:29:39.484841 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "130ea167f9b742ba94e63909a158e8b7"
format_stamp: "Formatted at 2026-09-23 19:29:39 on dist-test-slave-kxfb"
server_key: "08985e067fc9da42b0bddd0385c4b0c0"
server_key_iv: "06f3b5e5daa472bdd7217f70c68b313a"
server_key_version: "encryptionkey@0"
I20260923 19:29:39.490900 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 19:29:39.494895 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:39.496577 4177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:29:39.496767 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "130ea167f9b742ba94e63909a158e8b7"
format_stamp: "Formatted at 2026-09-23 19:29:39 on dist-test-slave-kxfb"
server_key: "08985e067fc9da42b0bddd0385c4b0c0"
server_key_iv: "06f3b5e5daa472bdd7217f70c68b313a"
server_key_version: "encryptionkey@0"
I20260923 19:29:39.496984 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:39.516430 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:39.520462 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:39.520727 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:39.521739 4177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:39.523169 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:39.523257 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:39.523373 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:39.523424 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:39.556951 4177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38483
I20260923 19:29:39.557058 4304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38483 every 8 connection(s)
I20260923 19:29:39.558251 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:39.566542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 4177
I20260923 19:29:39.566782 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:39.567409 388 external_mini_cluster.cc:1428] Setting key 22b2742c55e3f0689a97f729afee9aea
Sep 23 19:29:39 dist-test-slave-kxfb krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:39.581300 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:39.561194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34843 (local address 127.0.97.62:33907)
0923 19:29:39.561738 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:39.561746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:39.563022 (+ 1276us) server_negotiation.cc:407] Connection header received
0923 19:29:39.564957 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:39.564962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:39.565055 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:39.565197 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:39.566994 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:39.568158 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:39.570188 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:39.570700 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:39.570818 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:39.574890 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:39.574896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:39.574900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:39.574938 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:39.577610 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:39.578576 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:39.578580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:39.578583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:39.578720 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:39.579172 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:39.579175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:39.579177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:39.579480 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:39.579628 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:39.580481 (+ 853us) server_negotiation.cc:299] Negotiation successful
0923 19:29:39.580741 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":242,"threads_started":1}
I20260923 19:29:39.582445 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:39.561895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33907 (local address 127.0.97.2:34843)
0923 19:29:39.562821 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:39.562881 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:39.564619 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:39.565462 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:39.565472 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:39.566037 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:39.566767 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:39.566795 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:39.568369 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:39.568375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:39.570038 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:39.570056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:39.570952 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:39.570960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:39.571222 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:39.572053 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:39.572088 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:39.574610 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:39.577799 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:39.577812 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:39.577832 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:39.578439 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:39.578830 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:39.578835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:39.578839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:39.579057 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:39.579628 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:39.579638 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:39.580071 (+ 433us) client_negotiation.cc:769] Sending connection context
0923 19:29:39.580354 (+ 283us) client_negotiation.cc:240] Negotiation successful
0923 19:29:39.581005 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":374,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:29:39.584273 4305 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33907
I20260923 19:29:39.584646 4305 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:39.585556 4305 heartbeater.cc:511] Master 127.0.97.62:33907 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 19:29:39.587334 3982 ts_manager.cc:194] Registered new tserver with Master: 130ea167f9b742ba94e63909a158e8b7 (127.0.97.2:38483)
I20260923 19:29:39.588222 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:29:39.613440 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33907
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:39.942222 4311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:39.942543 4311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:39.942658 4311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:39.949666 4311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:39.949781 4311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:39.949918 4311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:39.959425 4311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:39.960680 4311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:39.961948 4311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:39.976019 4317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:39.976127 4319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:39.976291 4316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:39.977443 4311 server_base.cc:1034] running on GCE node
I20260923 19:29:39.978099 4311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:39.979135 4311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:39.980337 4311 hybrid_clock.cc:648] HybridClock initialized: now 1790191779980277 us; error 60 us; skew 500 ppm
I20260923 19:29:39.980630 4311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:39 dist-test-slave-kxfb krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:39.983997 4311 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:29:39.985500 4311 webserver.cc:460] Webserver started at http://127.0.97.3:46741/ using document root <none> and password file <none>
I20260923 19:29:39.986150 4311 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:39.986259 4311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:39.986572 4311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:39.989326 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "fe9e4862528e4e759a316937ad5a19da"
format_stamp: "Formatted at 2026-09-23 19:29:39 on dist-test-slave-kxfb"
server_key: "dbc443d6bb291b1bf9e0ca5869d09146"
server_key_iv: "db109f449da081a302542c4dfe5d512d"
server_key_version: "encryptionkey@0"
I20260923 19:29:39.990034 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fe9e4862528e4e759a316937ad5a19da"
format_stamp: "Formatted at 2026-09-23 19:29:39 on dist-test-slave-kxfb"
server_key: "dbc443d6bb291b1bf9e0ca5869d09146"
server_key_iv: "db109f449da081a302542c4dfe5d512d"
server_key_version: "encryptionkey@0"
I20260923 19:29:39.995282 4311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 19:29:39.998876 4326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:40.000385 4311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 19:29:40.000562 4311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "fe9e4862528e4e759a316937ad5a19da"
format_stamp: "Formatted at 2026-09-23 19:29:39 on dist-test-slave-kxfb"
server_key: "dbc443d6bb291b1bf9e0ca5869d09146"
server_key_iv: "db109f449da081a302542c4dfe5d512d"
server_key_version: "encryptionkey@0"
I20260923 19:29:40.000766 4311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:40.017154 4311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:40.022367 4311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:40.022675 4311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:40.023782 4311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:40.025327 4311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:40.025404 4311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:40.025511 4311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:40.025570 4311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:40.053737 4171 heartbeater.cc:503] Master 127.0.97.62:33907 was elected leader, sending a full tablet report...
I20260923 19:29:40.058099 4311 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35071
I20260923 19:29:40.058122 4438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35071 every 8 connection(s)
I20260923 19:29:40.059619 4311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:40.064754 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 4311
I20260923 19:29:40.064944 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:40.065521 388 external_mini_cluster.cc:1428] Setting key f1ee69fc91033131d3cae07243fabb6c
Sep 23 19:29:40 dist-test-slave-kxfb krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:40.082672 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:40.062272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34967 (local address 127.0.97.62:33907)
0923 19:29:40.062498 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:40.062503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:40.063904 (+ 1401us) server_negotiation.cc:407] Connection header received
0923 19:29:40.066574 (+ 2670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:40.066579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:40.066663 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:40.066781 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:40.068856 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.069707 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.071564 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.071976 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.072082 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:40.076050 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:40.076055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:40.076059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:40.076100 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:40.078740 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.079706 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.079710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.079713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.079832 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.080497 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.080503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.080510 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.080880 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:40.081031 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:40.082134 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0923 19:29:40.082413 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 19:29:40.084075 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:40.062726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33907 (local address 127.0.97.3:34967)
0923 19:29:40.063706 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:40.063757 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:40.066211 (+ 2454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:40.067062 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:40.067075 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:40.067861 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:40.068658 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:40.068678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.069892 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.069897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:40.071375 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:40.071389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.072209 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.072214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:40.072447 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:40.073178 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:40.073204 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:40.075756 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:40.078918 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:40.078927 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:40.078948 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:40.079537 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:40.079964 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:40.079970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:40.079974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:40.080306 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:40.081058 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:40.081068 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:40.081656 (+ 588us) client_negotiation.cc:769] Sending connection context
0923 19:29:40.081978 (+ 322us) client_negotiation.cc:240] Negotiation successful
0923 19:29:40.082639 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":404,"threads_started":1}
I20260923 19:29:40.085606 4439 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33907
I20260923 19:29:40.085920 4439 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:40.086728 4439 heartbeater.cc:511] Master 127.0.97.62:33907 requested a full tablet report, sending...
I20260923 19:29:40.088043 3982 ts_manager.cc:194] Registered new tserver with Master: fe9e4862528e4e759a316937ad5a19da (127.0.97.3:35071)
I20260923 19:29:40.088815 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34967
I20260923 19:29:40.095063 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 19:29:40 dist-test-slave-kxfb krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 19:29:40 dist-test-slave-kxfb krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:40.148460 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:40.132746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.0.97.1:34171)
0923 19:29:40.133516 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:40.133523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:40.133552 (+ 29us) server_negotiation.cc:407] Connection header received
0923 19:29:40.133615 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:40.133622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:40.133855 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:40.133991 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:40.135447 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.136430 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.137767 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.138371 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.138478 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:40.142673 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:40.142686 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:40.142701 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:40.142744 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:40.145596 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.146388 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.146398 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.146400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.146531 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.146932 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.146935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.146937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.147270 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:40.147501 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:40.147904 (+ 403us) server_negotiation.cc:299] Negotiation successful
0923 19:29:40.148124 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":421,"threads_started":1}
Sep 23 19:29:40 dist-test-slave-kxfb krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:40.167098 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:40.153217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.0.97.62:33907)
0923 19:29:40.153441 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:40.153448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:40.153544 (+ 96us) server_negotiation.cc:407] Connection header received
0923 19:29:40.153722 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:40.153729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:40.153804 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:40.153889 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:40.155232 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.156029 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.157269 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.157720 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.157825 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:40.161784 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:40.161789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:40.161793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:40.161829 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:40.164411 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.165038 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.165044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.165048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.165171 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.165560 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.165565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.165567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.165923 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:40.166180 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:40.166513 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 19:29:40.166758 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260923 19:29:40.168012 3982 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56674
I20260923 19:29:40.169162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 4043
I20260923 19:29:40.181692 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:40.170142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.0.97.1:34171)
0923 19:29:40.170396 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:40.170401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:40.170837 (+ 436us) server_negotiation.cc:407] Connection header received
0923 19:29:40.170993 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:40.170997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:40.171074 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:40.171158 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:40.172337 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.173061 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.174237 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.174718 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.174812 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:40.176898 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:40.176903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:40.176906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:40.176931 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:40.179438 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.180100 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.180103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.180106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.180231 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.180528 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.180531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.180533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.180795 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:40.180992 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:40.181224 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 19:29:40.181410 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 19:29:40.182528 4166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:40.271337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 4043
I20260923 19:29:40.287678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 4177
Sep 23 19:29:40 dist-test-slave-kxfb krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:29:40.304849 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:40.288729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.0.97.2:38483)
0923 19:29:40.289349 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:40.289359 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:40.289391 (+ 32us) server_negotiation.cc:407] Connection header received
0923 19:29:40.289455 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:40.289462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:40.289676 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:40.289825 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:40.291421 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.292450 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.293832 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.294437 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.294542 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:40.298952 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:40.298966 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:40.298982 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:40.299023 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:40.301969 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.302777 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.302783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.302786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.302901 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.303276 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.303280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.303282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.303648 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:40.303877 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:40.304284 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 19:29:40.304486 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":302,"threads_started":1}
I20260923 19:29:40.306640 4300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:40.393793 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 4177
I20260923 19:29:40.407271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 4311
Sep 23 19:29:40 dist-test-slave-kxfb krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:29:40.425168 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:40.408370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.0.97.3:35071)
0923 19:29:40.409104 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:40.409114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:40.409147 (+ 33us) server_negotiation.cc:407] Connection header received
0923 19:29:40.409208 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:40.409214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:40.409424 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:40.409555 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:40.410968 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.412065 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.413462 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:40.414022 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:40.414139 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:40.419121 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:40.419137 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:40.419158 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:40.419200 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:40.422101 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.422984 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.422993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.422996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.423155 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:40.423568 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:40.423571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:40.423573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:40.423972 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:40.424269 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:40.424581 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 19:29:40.424823 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":401,"threads_started":1}
I20260923 19:29:40.426790 4434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:40.515571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 4311
I20260923 19:29:40.529639 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 3950
I20260923 19:29:40.530979 4012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:40.646220 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 3950
2026-09-23T19:29:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3953 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:40 dist-test-slave-kxfb krb5kdc[4458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:40 dist-test-slave-kxfb krb5kdc[4458](info): set up 2 sockets
Sep 23 19:29:40 dist-test-slave-kxfb krb5kdc[4458](info): commencing operation
krb5kdc: starting...
W20260923 19:29:41.658653 388 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:41 dist-test-slave-kxfb krb5kdc[4458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:41.874174 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34767
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:42.240362 4471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:42.240742 4471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:42.240872 4471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:42.248761 4471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:42.248912 4471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:42.248973 4471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:42.249060 4471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:42.249130 4471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:42.259845 4471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:42.261482 4471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:42.263093 4471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:42.275884 4479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:42.275884 4477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:42.276265 4476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:42.276734 4471 server_base.cc:1034] running on GCE node
I20260923 19:29:42.277652 4471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:42.279367 4471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:42.280582 4471 hybrid_clock.cc:648] HybridClock initialized: now 1790191782280531 us; error 47 us; skew 500 ppm
I20260923 19:29:42.280964 4471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:42 dist-test-slave-kxfb krb5kdc[4458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:42.284297 4471 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:42.285665 4471 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:42.285774 4471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:42.286113 4471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:42.289615 4471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "2b97cf7c088044389fac8052ca27413d"
format_stamp: "Formatted at 2026-09-23 19:29:42 on dist-test-slave-kxfb"
server_key: "64ee88ad5376c2b98bb5d33ed7b41584"
server_key_iv: "005b8c285dff6c07f12b58e5711d67a6"
server_key_version: "encryptionkey@0"
I20260923 19:29:42.290540 4471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "2b97cf7c088044389fac8052ca27413d"
format_stamp: "Formatted at 2026-09-23 19:29:42 on dist-test-slave-kxfb"
server_key: "64ee88ad5376c2b98bb5d33ed7b41584"
server_key_iv: "005b8c285dff6c07f12b58e5711d67a6"
server_key_version: "encryptionkey@0"
I20260923 19:29:42.296314 4471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:29:42.300346 4485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:42.302109 4471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 19:29:42.302358 4471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "2b97cf7c088044389fac8052ca27413d"
format_stamp: "Formatted at 2026-09-23 19:29:42 on dist-test-slave-kxfb"
server_key: "64ee88ad5376c2b98bb5d33ed7b41584"
server_key_iv: "005b8c285dff6c07f12b58e5711d67a6"
server_key_version: "encryptionkey@0"
I20260923 19:29:42.302627 4471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:42.345680 4471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:42.350782 4471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:42.351082 4471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:42.370491 4471 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34767
I20260923 19:29:42.370524 4536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34767 every 8 connection(s)
I20260923 19:29:42.371510 4471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:42.374588 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 4471
I20260923 19:29:42.374780 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:42.375418 388 external_mini_cluster.cc:1428] Setting key 4ec4a287795ce893a19ff914fd9e3fae
I20260923 19:29:42.375658 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:42.389418 4537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d: Bootstrap starting.
Sep 23 19:29:42 dist-test-slave-kxfb krb5kdc[4458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:42.392777 4537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:42.394042 4537 log.cc:824] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:42.397433 4537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d: No bootstrap required, opened a new log
I20260923 19:29:42.399199 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:42.377756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.0.97.62:34767)
0923 19:29:42.378739 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:42.378765 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:42.378823 (+ 58us) server_negotiation.cc:407] Connection header received
0923 19:29:42.380587 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:42.380621 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:42.381130 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:42.381584 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:42.383071 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:42.384944 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:42.386270 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:42.387396 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:42.387582 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:42.391627 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:42.391637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:42.391656 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:42.391702 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:42.395299 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:42.396028 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:42.396034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:42.396041 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:42.396194 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:42.396532 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:42.396536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:42.396538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:42.397061 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:42.397329 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:42.397765 (+ 436us) server_negotiation.cc:299] Negotiation successful
0923 19:29:42.398164 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":446,"threads_started":1}
I20260923 19:29:42.406836 4537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97cf7c088044389fac8052ca27413d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34767 } }
I20260923 19:29:42.407291 4537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:42.407383 4537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b97cf7c088044389fac8052ca27413d, State: Initialized, Role: FOLLOWER
I20260923 19:29:42.408215 4537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97cf7c088044389fac8052ca27413d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34767 } }
I20260923 19:29:42.408569 4537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:42.408690 4537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:42.408841 4537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:42.411494 4537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97cf7c088044389fac8052ca27413d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34767 } }
I20260923 19:29:42.412015 4537 leader_election.cc:302] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b97cf7c088044389fac8052ca27413d; no voters:
I20260923 19:29:42.412636 4537 leader_election.cc:288] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:42.412787 4542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:42.413542 4542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [term 1 LEADER]: Becoming Leader. State: Replica: 2b97cf7c088044389fac8052ca27413d, State: Running, Role: LEADER
I20260923 19:29:42.414150 4542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97cf7c088044389fac8052ca27413d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34767 } }
I20260923 19:29:42.414999 4537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:42.417155 4544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b97cf7c088044389fac8052ca27413d. Latest consensus state: current_term: 1 leader_uuid: "2b97cf7c088044389fac8052ca27413d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97cf7c088044389fac8052ca27413d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34767 } } }
I20260923 19:29:42.417558 4544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:42.417933 4543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b97cf7c088044389fac8052ca27413d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b97cf7c088044389fac8052ca27413d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34767 } } }
I20260923 19:29:42.421558 4543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:42.422528 4548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:42.427062 4548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:42.436370 4548 catalog_manager.cc:1347] Generated new cluster ID: 89141fe51bcd4efda5c83768640b5a87
I20260923 19:29:42.436527 4548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:42.451112 4548 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:42.452404 4548 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:42.464939 4548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d: Generated new TSK 0
I20260923 19:29:42.465817 4548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:29:42.513826 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34767
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:42.851228 4563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:42.851512 4563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:42.851606 4563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:42.858939 4563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:42.859054 4563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:42.859167 4563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:42.868870 4563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:42.870110 4563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:42.871450 4563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:42.884003 4568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:42.884369 4569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:42.884840 4571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:42.885344 4563 server_base.cc:1034] running on GCE node
I20260923 19:29:42.885933 4563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:42.886932 4563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:42.888163 4563 hybrid_clock.cc:648] HybridClock initialized: now 1790191782888095 us; error 51 us; skew 500 ppm
I20260923 19:29:42.888481 4563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:42 dist-test-slave-kxfb krb5kdc[4458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:42.891443 4563 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:29:42.892686 4563 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:42.892786 4563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:42.893074 4563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:42.895949 4563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "5193ace6151b4ac9a8c829858dc2a400"
format_stamp: "Formatted at 2026-09-23 19:29:42 on dist-test-slave-kxfb"
server_key: "3ad0f8abceae88d32eb7fc9467e7eed3"
server_key_iv: "c9951a643688fbf33ec8dc4e8f56de09"
server_key_version: "encryptionkey@0"
I20260923 19:29:42.896763 4563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5193ace6151b4ac9a8c829858dc2a400"
format_stamp: "Formatted at 2026-09-23 19:29:42 on dist-test-slave-kxfb"
server_key: "3ad0f8abceae88d32eb7fc9467e7eed3"
server_key_iv: "c9951a643688fbf33ec8dc4e8f56de09"
server_key_version: "encryptionkey@0"
I20260923 19:29:42.901865 4563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:29:42.905685 4577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:42.907202 4563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:42.907388 4563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "5193ace6151b4ac9a8c829858dc2a400"
format_stamp: "Formatted at 2026-09-23 19:29:42 on dist-test-slave-kxfb"
server_key: "3ad0f8abceae88d32eb7fc9467e7eed3"
server_key_iv: "c9951a643688fbf33ec8dc4e8f56de09"
server_key_version: "encryptionkey@0"
I20260923 19:29:42.907588 4563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:42.937719 4563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:42.941717 4563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:42.941965 4563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:42.943011 4563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:42.944387 4563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:42.944474 4563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:42.944581 4563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:42.944659 4563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:42.977385 4563 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37103
I20260923 19:29:42.977479 4689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37103 every 8 connection(s)
I20260923 19:29:42.978268 4563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:42.983289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 4563
I20260923 19:29:42.983451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:42.984047 388 external_mini_cluster.cc:1428] Setting key 10fad281e484a2f9049dd6be4dcdc4f9
Sep 23 19:29:42 dist-test-slave-kxfb krb5kdc[4458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:43.000677 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:42.981171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39615 (local address 127.0.97.62:34767)
0923 19:29:42.981702 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:42.981709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:42.982825 (+ 1116us) server_negotiation.cc:407] Connection header received
0923 19:29:42.985024 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:42.985030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:42.985116 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:42.985232 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:42.987406 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:42.988462 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:42.990207 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:42.990789 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:42.990910 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:42.994960 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:42.994965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:42.994969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:42.995002 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:42.997358 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:42.998281 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:42.998309 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:42.998312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:42.998431 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:42.998868 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:42.998871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:42.998873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:42.999111 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:42.999213 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:43.000221 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0923 19:29:43.000402 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":254,"threads_started":1}
I20260923 19:29:43.002424 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:42.981701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34767 (local address 127.0.97.1:39615)
0923 19:29:42.982617 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:42.982677 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:42.984672 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:42.985607 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:42.985621 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:42.986495 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:42.987185 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:42.987206 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:42.988662 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:42.988666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:42.990047 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:42.990061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:42.991046 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:42.991052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:42.991311 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:42.992058 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:42.992083 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:42.994703 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:42.997500 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:42.997509 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:42.997531 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:42.998146 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:42.998528 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:42.998532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:42.998536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:42.998766 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:42.999227 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:42.999236 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:42.999726 (+ 490us) client_negotiation.cc:769] Sending connection context
0923 19:29:43.000066 (+ 340us) client_negotiation.cc:240] Negotiation successful
0923 19:29:43.000671 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":329,"threads_started":1}
I20260923 19:29:43.004233 4690 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34767
I20260923 19:29:43.004659 4690 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 19:29:43.005630 4690 heartbeater.cc:511] Master 127.0.97.62:34767 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 19:29:43.008072 4502 ts_manager.cc:194] Registered new tserver with Master: 5193ace6151b4ac9a8c829858dc2a400 (127.0.97.1:37103)
I20260923 19:29:43.009538 4502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:29:43.032006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34767
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:43.341631 4696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:43.341948 4696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:43.342097 4696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:43.349409 4696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:43.349541 4696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:43.349691 4696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:43.359333 4696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:43.360575 4696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:43.361804 4696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:43.374431 4702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:43.374537 4704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:43.375159 4701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:43.375880 4696 server_base.cc:1034] running on GCE node
I20260923 19:29:43.376469 4696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:43.377420 4696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:43.378625 4696 hybrid_clock.cc:648] HybridClock initialized: now 1790191783378553 us; error 72 us; skew 500 ppm
I20260923 19:29:43.378937 4696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:43 dist-test-slave-kxfb krb5kdc[4458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:43.385061 4696 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:29:43.386332 4696 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:43.386433 4696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:43.386736 4696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:43.389648 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "80430e10bacc40b9b3629310abd57a7a"
format_stamp: "Formatted at 2026-09-23 19:29:43 on dist-test-slave-kxfb"
server_key: "eb777c69691961457ea7348344816e92"
server_key_iv: "c7afad02841181a7c1e67633892da731"
server_key_version: "encryptionkey@0"
I20260923 19:29:43.390483 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "80430e10bacc40b9b3629310abd57a7a"
format_stamp: "Formatted at 2026-09-23 19:29:43 on dist-test-slave-kxfb"
server_key: "eb777c69691961457ea7348344816e92"
server_key_iv: "c7afad02841181a7c1e67633892da731"
server_key_version: "encryptionkey@0"
I20260923 19:29:43.395709 4696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 19:29:43.399516 4710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:43.401036 4696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:29:43.401202 4696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "80430e10bacc40b9b3629310abd57a7a"
format_stamp: "Formatted at 2026-09-23 19:29:43 on dist-test-slave-kxfb"
server_key: "eb777c69691961457ea7348344816e92"
server_key_iv: "c7afad02841181a7c1e67633892da731"
server_key_version: "encryptionkey@0"
I20260923 19:29:43.401414 4696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:43.417196 4696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:43.421211 4696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:43.421458 4696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:43.422441 4696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:43.423857 4696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:43.423944 4696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:43.424048 4696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:43.424124 4696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:43.457046 4696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35863
I20260923 19:29:43.457167 4822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35863 every 8 connection(s)
I20260923 19:29:43.457880 4696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:43.463199 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 4696
I20260923 19:29:43.463418 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:43.464015 388 external_mini_cluster.cc:1428] Setting key c15d564343334b6f548d1ea96eab44b8
Sep 23 19:29:43 dist-test-slave-kxfb krb5kdc[4458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:43.480397 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:43.460766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58783 (local address 127.0.97.62:34767)
0923 19:29:43.461020 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:43.461027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:43.462350 (+ 1323us) server_negotiation.cc:407] Connection header received
0923 19:29:43.464975 (+ 2625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:43.464981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:43.465073 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:43.465188 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:43.467180 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.467867 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.470112 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.470580 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.470681 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:43.474825 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:43.474832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:43.474836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:43.474868 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:43.477178 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:43.478050 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:43.478054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:43.478056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:43.478167 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:43.478618 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:43.478621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:43.478623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:43.478868 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:43.478979 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:43.479949 (+ 970us) server_negotiation.cc:299] Negotiation successful
0923 19:29:43.480116 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 19:29:43.482028 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:43.461202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34767 (local address 127.0.97.2:58783)
0923 19:29:43.462099 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:43.462161 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:43.464647 (+ 2486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:43.465495 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:43.465506 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:43.466087 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:43.466989 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:43.467014 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.468033 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.468040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:43.469960 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:43.469974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.470812 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.470820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:43.471090 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:43.471847 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:43.471880 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:43.474582 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:43.477317 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:43.477327 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:43.477345 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:43.477928 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:43.478261 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:43.478265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:43.478269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:43.478519 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:43.478987 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:43.478996 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:43.479495 (+ 499us) client_negotiation.cc:769] Sending connection context
0923 19:29:43.479858 (+ 363us) client_negotiation.cc:240] Negotiation successful
0923 19:29:43.480463 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":382,"threads_started":1}
I20260923 19:29:43.483911 4823 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34767
I20260923 19:29:43.484333 4823 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:43.485311 4823 heartbeater.cc:511] Master 127.0.97.62:34767 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:29:43.487238 4502 ts_manager.cc:194] Registered new tserver with Master: 80430e10bacc40b9b3629310abd57a7a (127.0.97.2:35863)
I20260923 19:29:43.488088 4502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:29:43.513456 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34767
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:43.826148 4828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:43.826464 4828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:43.826583 4828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:43.833716 4828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:43.833827 4828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:43.833973 4828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:43.843567 4828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:43.844826 4828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:43.846028 4828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:43.858536 4833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:43.858692 4834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:43.858989 4836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:43.860255 4828 server_base.cc:1034] running on GCE node
I20260923 19:29:43.860838 4828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:43.861793 4828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:43.863041 4828 hybrid_clock.cc:648] HybridClock initialized: now 1790191783862983 us; error 43 us; skew 500 ppm
I20260923 19:29:43.863363 4828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:43 dist-test-slave-kxfb krb5kdc[4458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:43.866389 4828 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:29:43.867549 4828 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:43.867640 4828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:43.867924 4828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:43.870775 4828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "339f99542b8648218911c2c8c431e565"
format_stamp: "Formatted at 2026-09-23 19:29:43 on dist-test-slave-kxfb"
server_key: "e268a97220ec80b032db2557690032df"
server_key_iv: "f5f79fa728ed30cca9aecbd2e09faa8a"
server_key_version: "encryptionkey@0"
I20260923 19:29:43.871498 4828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "339f99542b8648218911c2c8c431e565"
format_stamp: "Formatted at 2026-09-23 19:29:43 on dist-test-slave-kxfb"
server_key: "e268a97220ec80b032db2557690032df"
server_key_iv: "f5f79fa728ed30cca9aecbd2e09faa8a"
server_key_version: "encryptionkey@0"
I20260923 19:29:43.876408 4828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:29:43.880061 4842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:43.881762 4828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 19:29:43.881933 4828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "339f99542b8648218911c2c8c431e565"
format_stamp: "Formatted at 2026-09-23 19:29:43 on dist-test-slave-kxfb"
server_key: "e268a97220ec80b032db2557690032df"
server_key_iv: "f5f79fa728ed30cca9aecbd2e09faa8a"
server_key_version: "encryptionkey@0"
I20260923 19:29:43.882165 4828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:43.895387 4828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:43.900137 4828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:43.900419 4828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:43.901602 4828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:43.903154 4828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:43.903244 4828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:43.903357 4828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:43.903430 4828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:43.938217 4828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41701
I20260923 19:29:43.938267 4954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41701 every 8 connection(s)
I20260923 19:29:43.939245 4828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:43.941754 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 4828
I20260923 19:29:43.941933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:43.942502 388 external_mini_cluster.cc:1428] Setting key c84283580ac6aa9a18f10f7d432a18f5
Sep 23 19:29:43 dist-test-slave-kxfb krb5kdc[4458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:43.961344 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:43.941945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47257 (local address 127.0.97.62:34767)
0923 19:29:43.942154 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:43.942160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:43.943713 (+ 1553us) server_negotiation.cc:407] Connection header received
0923 19:29:43.945783 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:43.945787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:43.945854 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:43.945941 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:43.947868 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.948638 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.950570 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.951005 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.951123 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:43.955060 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:43.955065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:43.955068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:43.955104 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:43.958106 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:43.958961 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:43.958965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:43.958968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:43.959096 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:43.959501 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:43.959504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:43.959507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:43.959770 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:43.959912 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:43.960908 (+ 996us) server_negotiation.cc:299] Negotiation successful
0923 19:29:43.961094 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 19:29:43.962484 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:43.942603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34767 (local address 127.0.97.3:47257)
0923 19:29:43.943518 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:43.943587 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:43.945487 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:43.946256 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:43.946267 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:43.946973 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:43.947685 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:43.947706 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.948822 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.948827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:43.950337 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:43.950356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.951255 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.951260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:43.951487 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:43.952272 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:43.952302 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:43.954797 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:43.958259 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:43.958268 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:43.958328 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:43.958834 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:43.959185 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:43.959189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:43.959191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:43.959379 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:43.959904 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:43.959911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:43.960341 (+ 430us) client_negotiation.cc:769] Sending connection context
0923 19:29:43.960615 (+ 274us) client_negotiation.cc:240] Negotiation successful
0923 19:29:43.961101 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":376,"threads_started":1}
I20260923 19:29:43.964062 4955 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34767
I20260923 19:29:43.964389 4955 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:43.965101 4955 heartbeater.cc:511] Master 127.0.97.62:34767 requested a full tablet report, sending...
I20260923 19:29:43.966677 4502 ts_manager.cc:194] Registered new tserver with Master: 339f99542b8648218911c2c8c431e565 (127.0.97.3:41701)
I20260923 19:29:43.967577 4502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47257
I20260923 19:29:43.970353 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:29:43.987320 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:43.975322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.0.97.62:34767)
0923 19:29:43.975526 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:43.975533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:43.975982 (+ 449us) server_negotiation.cc:407] Connection header received
0923 19:29:43.976161 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:43.976165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:43.976228 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:43.976308 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:43.977762 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.978528 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.979733 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:43.980170 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:43.980278 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:43.982615 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:43.982620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:43.982623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:43.982653 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:43.984884 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:43.985442 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:43.985447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:43.985450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:43.985573 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:43.985902 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:43.985906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:43.985908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:43.986229 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:43.986479 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:43.986771 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 19:29:43.986990 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 19:29:43.992194 4502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:29:43.999845 4502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:44.012596 4690 heartbeater.cc:503] Master 127.0.97.62:34767 was elected leader, sending a full tablet report...
I20260923 19:29:44.040038 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.016113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.0.97.2:35863)
0923 19:29:44.030941 (+ 14828us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:44.030954 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:44.031758 (+ 804us) server_negotiation.cc:407] Connection header received
0923 19:29:44.031910 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:44.031921 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:44.032299 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:44.032523 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:44.033602 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.034849 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.037750 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.038810 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.038931 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.039134 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:44.039524 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 19:29:44.039682 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14610,"thread_start_us":14436,"threads_started":1}
I20260923 19:29:44.041914 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.015563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35863 (local address 127.0.0.1:39472)
0923 19:29:44.031456 (+ 15893us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:44.031475 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:44.031630 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:44.032889 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:44.032899 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:44.032943 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:44.033354 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.033367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.035047 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.035056 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.037593 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.037606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.039021 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.039028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.039258 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.039311 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 19:29:44.039447 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 19:29:44.041491 (+ 2044us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15744,"mutex_wait_us":1798,"thread_start_us":15591,"threads_started":1}
I20260923 19:29:44.042366 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.015209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37103 (local address 127.0.0.1:46796)
0923 19:29:44.029600 (+ 14391us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:44.029624 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:44.029759 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:44.034942 (+ 5183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:44.034948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:44.034986 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:44.035481 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.035497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.037416 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.037421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.039596 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.039611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.041209 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.041215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.041472 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.041506 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:29:44.041594 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 19:29:44.041733 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14223,"mutex_wait_us":27,"thread_start_us":11085,"threads_started":1}
I20260923 19:29:44.042518 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.015912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.0.97.1:37103)
0923 19:29:44.034107 (+ 18195us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:44.034119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:44.034156 (+ 37us) server_negotiation.cc:407] Connection header received
0923 19:29:44.034271 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:44.034281 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:44.034665 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:44.034933 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:44.035747 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.037270 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.039769 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.041001 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.041139 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.041359 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:44.042016 (+ 657us) server_negotiation.cc:299] Negotiation successful
0923 19:29:44.042178 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17970,"thread_start_us":17758,"threads_started":1}
I20260923 19:29:44.044165 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.015115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41701 (local address 127.0.0.1:44140)
0923 19:29:44.026968 (+ 11853us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:44.027065 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:44.027341 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:44.036606 (+ 9265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:44.036612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:44.036638 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:44.037065 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.037075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.039088 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.039094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.041405 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.041417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.043382 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.043388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.043558 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.043589 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 19:29:44.043682 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 19:29:44.043819 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11433,"mutex_wait_us":29,"thread_start_us":14726,"threads_started":1}
I20260923 19:29:44.045409 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.015379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.0.97.3:41701)
0923 19:29:44.035661 (+ 20282us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:44.035670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:44.035705 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:29:44.035809 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:44.035817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:44.036056 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:44.036266 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:44.037246 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.038894 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.041561 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.042930 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.043060 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.043272 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:44.044491 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0923 19:29:44.044702 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20065,"thread_start_us":21210,"threads_started":1}
I20260923 19:29:44.046073 4625 tablet_service.cc:1467] Processing CreateTablet for tablet ef669ff1705f477b8d8004b81698e9c1 (DEFAULT_TABLE table=test-table [id=e3cfefc24b7d4cf389d74ad568d1acfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:44.046725 4758 tablet_service.cc:1467] Processing CreateTablet for tablet ef669ff1705f477b8d8004b81698e9c1 (DEFAULT_TABLE table=test-table [id=e3cfefc24b7d4cf389d74ad568d1acfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:44.047359 4625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef669ff1705f477b8d8004b81698e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:44.048077 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef669ff1705f477b8d8004b81698e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:44.056381 4890 tablet_service.cc:1467] Processing CreateTablet for tablet ef669ff1705f477b8d8004b81698e9c1 (DEFAULT_TABLE table=test-table [id=e3cfefc24b7d4cf389d74ad568d1acfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:44.057896 4890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef669ff1705f477b8d8004b81698e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:44.060382 4973 tablet_bootstrap.cc:492] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: Bootstrap starting.
I20260923 19:29:44.065171 4973 tablet_bootstrap.cc:654] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:44.066255 4974 tablet_bootstrap.cc:492] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: Bootstrap starting.
I20260923 19:29:44.066784 4973 log.cc:824] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:44.069996 4973 tablet_bootstrap.cc:492] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: No bootstrap required, opened a new log
I20260923 19:29:44.070235 4974 tablet_bootstrap.cc:654] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:44.070355 4973 ts_tablet_manager.cc:1397] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260923 19:29:44.071748 4974 log.cc:824] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:44.079329 4976 tablet_bootstrap.cc:492] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: Bootstrap starting.
I20260923 19:29:44.079113 4973 raft_consensus.cc:348] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } }
I20260923 19:29:44.080274 4973 raft_consensus.cc:374] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:44.080440 4973 raft_consensus.cc:729] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5193ace6151b4ac9a8c829858dc2a400, State: Initialized, Role: FOLLOWER
I20260923 19:29:44.081373 4973 consensus_queue.cc:260] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } }
I20260923 19:29:44.082589 4974 tablet_bootstrap.cc:492] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: No bootstrap required, opened a new log
I20260923 19:29:44.082944 4974 ts_tablet_manager.cc:1397] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20260923 19:29:44.083529 4973 ts_tablet_manager.cc:1428] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260923 19:29:44.085342 4976 tablet_bootstrap.cc:654] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:44.086635 4976 log.cc:824] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:44.090446 4976 tablet_bootstrap.cc:492] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: No bootstrap required, opened a new log
I20260923 19:29:44.090756 4976 ts_tablet_manager.cc:1397] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260923 19:29:44.092531 4974 raft_consensus.cc:348] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } }
I20260923 19:29:44.092998 4974 raft_consensus.cc:374] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:44.093098 4974 raft_consensus.cc:729] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80430e10bacc40b9b3629310abd57a7a, State: Initialized, Role: FOLLOWER
I20260923 19:29:44.093742 4974 consensus_queue.cc:260] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } }
I20260923 19:29:44.095068 4823 heartbeater.cc:503] Master 127.0.97.62:34767 was elected leader, sending a full tablet report...
I20260923 19:29:44.095664 4974 ts_tablet_manager.cc:1428] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260923 19:29:44.100306 4976 raft_consensus.cc:348] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } }
I20260923 19:29:44.100731 4976 raft_consensus.cc:374] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:44.100823 4976 raft_consensus.cc:729] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339f99542b8648218911c2c8c431e565, State: Initialized, Role: FOLLOWER
I20260923 19:29:44.101357 4976 consensus_queue.cc:260] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } }
I20260923 19:29:44.102494 4955 heartbeater.cc:503] Master 127.0.97.62:34767 was elected leader, sending a full tablet report...
I20260923 19:29:44.103066 4976 ts_tablet_manager.cc:1428] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260923 19:29:44.157949 4978 raft_consensus.cc:482] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:44.158257 4978 raft_consensus.cc:504] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } }
I20260923 19:29:44.159992 4978 leader_election.cc:288] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 339f99542b8648218911c2c8c431e565 (127.0.97.3:41701), 80430e10bacc40b9b3629310abd57a7a (127.0.97.2:35863)
I20260923 19:29:44.167331 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.160586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41149 (local address 127.0.97.2:35863)
0923 19:29:44.160846 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:44.160853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:44.161867 (+ 1014us) server_negotiation.cc:407] Connection header received
0923 19:29:44.162060 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:44.162064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:44.162128 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:44.162204 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:44.162995 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.163691 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.165627 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.166439 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.166560 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.166658 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:44.166964 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 19:29:44.167084 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 19:29:44.167572 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.160647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35719 (local address 127.0.97.3:41701)
0923 19:29:44.160903 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:44.160911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:44.161819 (+ 908us) server_negotiation.cc:407] Connection header received
0923 19:29:44.162036 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:44.162042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:44.162112 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:44.162193 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:29:44.162919 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.163709 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.165735 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.166611 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.166731 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.166851 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:44.167151 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 19:29:44.167289 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 19:29:44.167622 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.160511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35863 (local address 127.0.97.1:41149)
0923 19:29:44.161737 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:44.161755 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:44.161918 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:44.162239 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:44.162243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:44.162373 (+ 130us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:44.162819 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.162830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.163825 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.163830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.165454 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.165468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.166639 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.166646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.166812 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.166842 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 19:29:44.166923 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 19:29:44.167045 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"spinlock_wait_cycles":104192,"thread_start_us":654,"threads_started":1}
I20260923 19:29:44.167788 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.160357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41701 (local address 127.0.97.1:35719)
0923 19:29:44.161677 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:44.161695 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:44.161886 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:44.162224 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:44.162229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:44.162258 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:29:44.162764 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.162779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.163841 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.163846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.165571 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:44.165584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.166851 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.166857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:44.166992 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.167024 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 19:29:44.167106 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 19:29:44.167260 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":743,"threads_started":1}
I20260923 19:29:44.168841 4778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef669ff1705f477b8d8004b81698e9c1" candidate_uuid: "5193ace6151b4ac9a8c829858dc2a400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80430e10bacc40b9b3629310abd57a7a" is_pre_election: true
I20260923 19:29:44.169332 4778 raft_consensus.cc:2454] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5193ace6151b4ac9a8c829858dc2a400 in term 0.
I20260923 19:29:44.169420 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef669ff1705f477b8d8004b81698e9c1" candidate_uuid: "5193ace6151b4ac9a8c829858dc2a400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339f99542b8648218911c2c8c431e565" is_pre_election: true
I20260923 19:29:44.170017 4581 leader_election.cc:302] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5193ace6151b4ac9a8c829858dc2a400, 80430e10bacc40b9b3629310abd57a7a; no voters:
I20260923 19:29:44.170042 4910 raft_consensus.cc:2454] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5193ace6151b4ac9a8c829858dc2a400 in term 0.
I20260923 19:29:44.170450 4978 raft_consensus.cc:2781] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:44.170606 4978 raft_consensus.cc:482] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:44.170722 4978 raft_consensus.cc:3037] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:44.172928 4978 raft_consensus.cc:504] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } }
I20260923 19:29:44.173486 4978 leader_election.cc:288] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [CANDIDATE]: Term 1 election: Requested vote from peers 339f99542b8648218911c2c8c431e565 (127.0.97.3:41701), 80430e10bacc40b9b3629310abd57a7a (127.0.97.2:35863)
I20260923 19:29:44.174070 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef669ff1705f477b8d8004b81698e9c1" candidate_uuid: "5193ace6151b4ac9a8c829858dc2a400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339f99542b8648218911c2c8c431e565"
I20260923 19:29:44.174149 4778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef669ff1705f477b8d8004b81698e9c1" candidate_uuid: "5193ace6151b4ac9a8c829858dc2a400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80430e10bacc40b9b3629310abd57a7a"
I20260923 19:29:44.174391 4778 raft_consensus.cc:3037] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:44.174371 4910 raft_consensus.cc:3037] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:44.177500 4778 raft_consensus.cc:2454] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5193ace6151b4ac9a8c829858dc2a400 in term 1.
I20260923 19:29:44.177500 4910 raft_consensus.cc:2454] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5193ace6151b4ac9a8c829858dc2a400 in term 1.
I20260923 19:29:44.177980 4580 leader_election.cc:302] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 339f99542b8648218911c2c8c431e565, 5193ace6151b4ac9a8c829858dc2a400, 80430e10bacc40b9b3629310abd57a7a; no voters:
I20260923 19:29:44.178385 4978 raft_consensus.cc:2781] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:44.178944 4978 raft_consensus.cc:686] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 1 LEADER]: Becoming Leader. State: Replica: 5193ace6151b4ac9a8c829858dc2a400, State: Running, Role: LEADER
I20260923 19:29:44.179543 4978 consensus_queue.cc:237] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } }
I20260923 19:29:44.184756 4502 catalog_manager.cc:5495] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5193ace6151b4ac9a8c829858dc2a400 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5193ace6151b4ac9a8c829858dc2a400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5193ace6151b4ac9a8c829858dc2a400" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 } health_report { overall_health: UNKNOWN } } }
W20260923 19:29:44.190867 4956 tablet.cc:2369] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:29:44.209844 4824 tablet.cc:2369] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:29:44.230592 4691 tablet.cc:2369] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:29:44.238540 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.233079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.0.97.1:37103)
0923 19:29:44.233309 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:44.233315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:44.233445 (+ 130us) server_negotiation.cc:407] Connection header received
0923 19:29:44.233679 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:44.233683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:44.233769 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:44.233902 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:29:44.234267 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.235132 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.236456 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.236829 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.236933 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.237402 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:29:44.237943 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:29:44.238067 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:44.238168 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 19:29:44.238246 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 19:29:44.247849 4778 raft_consensus.cc:1261] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [term 1 FOLLOWER]: Refusing update from remote peer 5193ace6151b4ac9a8c829858dc2a400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:44.247853 4910 raft_consensus.cc:1261] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [term 1 FOLLOWER]: Refusing update from remote peer 5193ace6151b4ac9a8c829858dc2a400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:44.249120 4984 consensus_queue.cc:1035] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80430e10bacc40b9b3629310abd57a7a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:44.249578 4978 consensus_queue.cc:1035] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "339f99542b8648218911c2c8c431e565" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:44.263417 4987 mvcc.cc:205] Tried to move back new op lower bound from 7332625548265963520 to 7332625548003205120. Current Snapshot: MvccSnapshot[applied={T|T < 7332625548265963520}]
I20260923 19:29:44.279245 4501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:44.279601 4501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:44.284078 4501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b97cf7c088044389fac8052ca27413d: Sending DeleteTablet for 3 replicas of tablet ef669ff1705f477b8d8004b81698e9c1
I20260923 19:29:44.285501 4890 tablet_service.cc:1514] Processing DeleteTablet for tablet ef669ff1705f477b8d8004b81698e9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44140
I20260923 19:29:44.285496 4625 tablet_service.cc:1514] Processing DeleteTablet for tablet ef669ff1705f477b8d8004b81698e9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46796
I20260923 19:29:44.285665 4758 tablet_service.cc:1514] Processing DeleteTablet for tablet ef669ff1705f477b8d8004b81698e9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39472
I20260923 19:29:44.287134 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 4563
I20260923 19:29:44.298218 4999 tablet_replica.cc:330] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: stopping tablet replica
I20260923 19:29:44.299027 4999 raft_consensus.cc:2229] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:29:44.299984 4999 raft_consensus.cc:2258] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:44.300973 4998 tablet_replica.cc:330] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: stopping tablet replica
I20260923 19:29:44.301327 4997 tablet_replica.cc:330] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: stopping tablet replica
I20260923 19:29:44.301429 4998 raft_consensus.cc:2229] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:44.301805 4997 raft_consensus.cc:2229] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:44.302053 4998 raft_consensus.cc:2258] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:44.302363 4997 raft_consensus.cc:2258] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:44.304795 4997 ts_tablet_manager.cc:1905] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:44.304806 4999 ts_tablet_manager.cc:1905] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:44.305212 4998 ts_tablet_manager.cc:1905] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:44.313124 4997 ts_tablet_manager.cc:1918] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:29:44.313284 4999 ts_tablet_manager.cc:1918] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:29:44.313463 4999 log.cc:1196] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/ef669ff1705f477b8d8004b81698e9c1
I20260923 19:29:44.313468 4997 log.cc:1196] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/ef669ff1705f477b8d8004b81698e9c1
I20260923 19:29:44.314013 4997 ts_tablet_manager.cc:1939] T ef669ff1705f477b8d8004b81698e9c1 P 339f99542b8648218911c2c8c431e565: Deleting consensus metadata
I20260923 19:29:44.314236 4999 ts_tablet_manager.cc:1939] T ef669ff1705f477b8d8004b81698e9c1 P 5193ace6151b4ac9a8c829858dc2a400: Deleting consensus metadata
I20260923 19:29:44.316349 4486 catalog_manager.cc:4841] TS 5193ace6151b4ac9a8c829858dc2a400 (127.0.97.1:37103): tablet ef669ff1705f477b8d8004b81698e9c1 (table test-table [id=e3cfefc24b7d4cf389d74ad568d1acfa]) successfully deleted
I20260923 19:29:44.316395 4488 catalog_manager.cc:4841] TS 339f99542b8648218911c2c8c431e565 (127.0.97.3:41701): tablet ef669ff1705f477b8d8004b81698e9c1 (table test-table [id=e3cfefc24b7d4cf389d74ad568d1acfa]) successfully deleted
I20260923 19:29:44.320066 4998 ts_tablet_manager.cc:1918] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:29:44.320259 4998 log.cc:1196] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/ef669ff1705f477b8d8004b81698e9c1
I20260923 19:29:44.320762 4998 ts_tablet_manager.cc:1939] T ef669ff1705f477b8d8004b81698e9c1 P 80430e10bacc40b9b3629310abd57a7a: Deleting consensus metadata
Sep 23 19:29:44 dist-test-slave-kxfb krb5kdc[4458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:44.322721 4489 catalog_manager.cc:4841] TS 80430e10bacc40b9b3629310abd57a7a (127.0.97.2:35863): tablet ef669ff1705f477b8d8004b81698e9c1 (table test-table [id=e3cfefc24b7d4cf389d74ad568d1acfa]) successfully deleted
I20260923 19:29:44.330212 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.288365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.0.97.1:37103)
0923 19:29:44.288567 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:44.288573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:44.308977 (+ 20404us) server_negotiation.cc:407] Connection header received
0923 19:29:44.310705 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:44.310711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:44.310837 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:44.310920 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:44.314414 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.315676 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.317507 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.318271 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.318479 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.323932 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:44.323947 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:44.323970 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:44.324018 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:44.327272 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:44.328009 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:44.328017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:44.328020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:44.328191 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:44.328555 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:44.328559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:44.328561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:44.328974 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:44.329287 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:44.329564 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 19:29:44.329820 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 19:29:44.331213 4685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:44.442978 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 4563
I20260923 19:29:44.459537 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 4696
Sep 23 19:29:44 dist-test-slave-kxfb krb5kdc[4458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:29:44.477784 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.460430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.0.97.2:35863)
0923 19:29:44.460662 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:44.460666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:44.460692 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:29:44.460914 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:44.460918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:44.461025 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:44.461118 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:44.462703 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.463591 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.465462 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.465970 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.466106 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.471182 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:44.471196 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:44.471216 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:44.471269 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:44.474973 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:44.475737 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:44.475752 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:44.475756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:44.475893 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:44.476204 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:44.476209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:44.476212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:44.476673 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:44.476998 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:44.477199 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 19:29:44.477414 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 19:29:44.478667 4818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:44.567780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 4696
I20260923 19:29:44.581723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 4828
Sep 23 19:29:44 dist-test-slave-kxfb krb5kdc[4458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:29:44.596293 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:44.582589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.0.97.3:41701)
0923 19:29:44.582800 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:44.582806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:44.582829 (+ 23us) server_negotiation.cc:407] Connection header received
0923 19:29:44.582980 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:44.582987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:44.583061 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:44.583159 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:44.584349 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.585088 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.586381 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:44.586872 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:44.586994 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:44.591266 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:44.591279 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:44.591297 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:44.591332 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:44.593854 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:44.594630 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:44.594636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:44.594639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:44.594752 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:44.595065 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:44.595068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:44.595070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:44.595378 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:44.595584 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:44.595861 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 19:29:44.596030 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 19:29:44.596997 4950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:44.686548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 4828
I20260923 19:29:44.699887 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 4471
I20260923 19:29:44.701220 4532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:44.806870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 4471
2026-09-23T19:29:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4156 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T19:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:44Z Disabled control of system clock
I20260923 19:29:44.876814 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45353
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 19:29:45.236146 5008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:45.236526 5008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:45.236657 5008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:45.244169 5008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:45.244320 5008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:45.244360 5008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:45.244424 5008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:45.244472 5008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:45.254232 5008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45353
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:45.255826 5008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:45.257920 5008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:45.270634 5014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:45.270773 5013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:45.270918 5016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:45.271929 5008 server_base.cc:1034] running on GCE node
I20260923 19:29:45.272825 5008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:45.274565 5008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:45.275796 5008 hybrid_clock.cc:648] HybridClock initialized: now 1790191785275718 us; error 66 us; skew 500 ppm
I20260923 19:29:45.276203 5008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:29:45.278959 5008 webserver.cc:460] Webserver started at http://127.0.97.62:34417/ using document root <none> and password file <none>
I20260923 19:29:45.279757 5008 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:45.279860 5008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:45.280215 5008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:45.283560 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "7c34020bbd3844e2a1e3da59bbdedfde"
format_stamp: "Formatted at 2026-09-23 19:29:45 on dist-test-slave-kxfb"
server_key: "304875179a83dae384475aca5f2d90b4"
server_key_iv: "f883a6281f89926fb98b1b56c2b37f61"
server_key_version: "encryptionkey@0"
I20260923 19:29:45.284456 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "7c34020bbd3844e2a1e3da59bbdedfde"
format_stamp: "Formatted at 2026-09-23 19:29:45 on dist-test-slave-kxfb"
server_key: "304875179a83dae384475aca5f2d90b4"
server_key_iv: "f883a6281f89926fb98b1b56c2b37f61"
server_key_version: "encryptionkey@0"
I20260923 19:29:45.290411 5008 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 19:29:45.294545 5022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:45.296147 5008 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:29:45.296325 5008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "7c34020bbd3844e2a1e3da59bbdedfde"
format_stamp: "Formatted at 2026-09-23 19:29:45 on dist-test-slave-kxfb"
server_key: "304875179a83dae384475aca5f2d90b4"
server_key_iv: "f883a6281f89926fb98b1b56c2b37f61"
server_key_version: "encryptionkey@0"
I20260923 19:29:45.296545 5008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:45.302762 5008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:45.303615 5008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:45.303903 5008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:45.323155 5008 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45353
I20260923 19:29:45.323194 5073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45353 every 8 connection(s)
I20260923 19:29:45.324684 5008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:45.328600 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 5008
I20260923 19:29:45.328771 5074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:45.328802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:45.329433 388 external_mini_cluster.cc:1428] Setting key 1a625f3db0a9f0c9ae6d70e07507ba9e
I20260923 19:29:45.339270 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:45.331869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.0.97.62:45353)
0923 19:29:45.333101 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:45.333123 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:45.333186 (+ 63us) server_negotiation.cc:407] Connection header received
0923 19:29:45.335171 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:45.335201 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:45.335792 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:45.336277 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:45.337258 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:45.337266 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:45.337281 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:45.337297 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:45.337373 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:45.337396 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:45.337626 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:45.337746 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 19:29:45.338034 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":363,"threads_started":1}
I20260923 19:29:45.342484 5074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde: Bootstrap starting.
I20260923 19:29:45.345614 5074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:45.346529 5074 log.cc:824] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:45.349406 5074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde: No bootstrap required, opened a new log
I20260923 19:29:45.356194 5074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34020bbd3844e2a1e3da59bbdedfde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } }
I20260923 19:29:45.356613 5074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:45.356725 5074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c34020bbd3844e2a1e3da59bbdedfde, State: Initialized, Role: FOLLOWER
I20260923 19:29:45.357347 5074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34020bbd3844e2a1e3da59bbdedfde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } }
I20260923 19:29:45.357584 5074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:45.357686 5074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:45.357798 5074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:45.359768 5074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34020bbd3844e2a1e3da59bbdedfde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } }
I20260923 19:29:45.360193 5074 leader_election.cc:302] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c34020bbd3844e2a1e3da59bbdedfde; no voters:
I20260923 19:29:45.360739 5074 leader_election.cc:288] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:45.360944 5079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:45.361876 5079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [term 1 LEADER]: Becoming Leader. State: Replica: 7c34020bbd3844e2a1e3da59bbdedfde, State: Running, Role: LEADER
I20260923 19:29:45.362608 5079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34020bbd3844e2a1e3da59bbdedfde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } }
I20260923 19:29:45.363021 5074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:45.365974 5081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c34020bbd3844e2a1e3da59bbdedfde. Latest consensus state: current_term: 1 leader_uuid: "7c34020bbd3844e2a1e3da59bbdedfde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34020bbd3844e2a1e3da59bbdedfde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } } }
I20260923 19:29:45.366451 5081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:45.366519 5080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c34020bbd3844e2a1e3da59bbdedfde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c34020bbd3844e2a1e3da59bbdedfde" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } } }
I20260923 19:29:45.367162 5080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:45.368861 5088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:45.374361 5088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:45.383538 5088 catalog_manager.cc:1347] Generated new cluster ID: 44156f8ab4d54f2c8e9c782c03a03772
I20260923 19:29:45.383754 5088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:45.411906 5088 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:45.412389 5088 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:45.421293 5088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde: Generated new TSK 0
I20260923 19:29:45.422516 5088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:29:45.439800 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45353
--builtin_ntp_servers=127.0.97.20:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 19:29:45.802904 5098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:45.803242 5098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:45.803340 5098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:45.811231 5098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:45.811354 5098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:45.811475 5098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:45.821125 5098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:45.822736 5098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:45.824792 5098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:45.839129 5103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:45.839453 5106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:45.840070 5104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:45.840576 5098 server_base.cc:1034] running on GCE node
I20260923 19:29:45.841261 5098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:45.842379 5098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:45.843590 5098 hybrid_clock.cc:648] HybridClock initialized: now 1790191785843537 us; error 48 us; skew 500 ppm
I20260923 19:29:45.843930 5098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:29:45.846513 5098 webserver.cc:460] Webserver started at http://127.0.97.1:41009/ using document root <none> and password file <none>
I20260923 19:29:45.847316 5098 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:45.847419 5098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:45.847787 5098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:45.850987 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "5fe8774e5a58499da5c812945db88933"
format_stamp: "Formatted at 2026-09-23 19:29:45 on dist-test-slave-kxfb"
server_key: "e838730be35e5d5534b94559cda1fe62"
server_key_iv: "29cfe25fd304ed0c28588eff08b75111"
server_key_version: "encryptionkey@0"
I20260923 19:29:45.851816 5098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5fe8774e5a58499da5c812945db88933"
format_stamp: "Formatted at 2026-09-23 19:29:45 on dist-test-slave-kxfb"
server_key: "e838730be35e5d5534b94559cda1fe62"
server_key_iv: "29cfe25fd304ed0c28588eff08b75111"
server_key_version: "encryptionkey@0"
I20260923 19:29:45.857493 5098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:29:45.861352 5112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:45.862978 5098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:45.863143 5098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "5fe8774e5a58499da5c812945db88933"
format_stamp: "Formatted at 2026-09-23 19:29:45 on dist-test-slave-kxfb"
server_key: "e838730be35e5d5534b94559cda1fe62"
server_key_iv: "29cfe25fd304ed0c28588eff08b75111"
server_key_version: "encryptionkey@0"
I20260923 19:29:45.863370 5098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:45.869751 5098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:45.870617 5098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:45.870869 5098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:45.871817 5098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:45.873317 5098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:45.873411 5098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:45.873520 5098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:45.873591 5098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:45.907052 5098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45997
I20260923 19:29:45.907120 5224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45997 every 8 connection(s)
I20260923 19:29:45.908588 5098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:45.911347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 5098
I20260923 19:29:45.911572 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:45.912374 388 external_mini_cluster.cc:1428] Setting key c2125921c974777f1e936f73e78bd448
I20260923 19:29:45.915472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45353
--builtin_ntp_servers=127.0.97.20:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 19:29:45.917917 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:45.911581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36277 (local address 127.0.97.62:45353)
0923 19:29:45.912032 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:45.912036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:45.913675 (+ 1639us) server_negotiation.cc:407] Connection header received
0923 19:29:45.915508 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:45.915514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:45.915577 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:45.915675 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:45.917156 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:45.917160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:45.917163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:45.917177 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:45.917215 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:45.917226 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:45.917343 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:45.917518 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 19:29:45.917591 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":252,"threads_started":1}
I20260923 19:29:45.918963 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:45.912392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45353 (local address 127.0.97.1:36277)
0923 19:29:45.913493 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:45.913542 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:45.915232 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:45.915969 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:45.915977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:45.916072 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:45.916845 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:29:45.916847 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:45.916889 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:29:45.916977 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:45.917324 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:45.917334 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:45.917341 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 19:29:45.917608 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 19:29:45.917894 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":327,"threads_started":1}
I20260923 19:29:45.920625 5225 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45353
I20260923 19:29:45.921007 5225 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:45.921811 5225 heartbeater.cc:511] Master 127.0.97.62:45353 requested a full tablet report, sending...
I20260923 19:29:45.924196 5039 ts_manager.cc:194] Registered new tserver with Master: 5fe8774e5a58499da5c812945db88933 (127.0.97.1:45997)
W20260923 19:29:46.251230 5229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:46.251530 5229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:46.251626 5229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:46.258880 5229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:46.258988 5229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:46.259116 5229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:46.268713 5229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:46.270054 5229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:46.271792 5229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:46.285596 5234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:46.285703 5235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:46.285863 5237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:46.286387 5229 server_base.cc:1034] running on GCE node
I20260923 19:29:46.286986 5229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:46.287945 5229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:46.289153 5229 hybrid_clock.cc:648] HybridClock initialized: now 1790191786289100 us; error 53 us; skew 500 ppm
I20260923 19:29:46.289471 5229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:29:46.291915 5229 webserver.cc:460] Webserver started at http://127.0.97.2:45557/ using document root <none> and password file <none>
I20260923 19:29:46.292614 5229 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:46.292711 5229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:46.293015 5229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:46.295948 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "576a5c665d7943b5951f9229175cc388"
format_stamp: "Formatted at 2026-09-23 19:29:46 on dist-test-slave-kxfb"
server_key: "d5a063584cb0643c82a7c8f22e2e4eff"
server_key_iv: "c8d8a0d088e6eec5621699cb84fe085b"
server_key_version: "encryptionkey@0"
I20260923 19:29:46.296696 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "576a5c665d7943b5951f9229175cc388"
format_stamp: "Formatted at 2026-09-23 19:29:46 on dist-test-slave-kxfb"
server_key: "d5a063584cb0643c82a7c8f22e2e4eff"
server_key_iv: "c8d8a0d088e6eec5621699cb84fe085b"
server_key_version: "encryptionkey@0"
I20260923 19:29:46.302188 5229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 19:29:46.305821 5243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:46.307416 5229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:46.307590 5229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "576a5c665d7943b5951f9229175cc388"
format_stamp: "Formatted at 2026-09-23 19:29:46 on dist-test-slave-kxfb"
server_key: "d5a063584cb0643c82a7c8f22e2e4eff"
server_key_iv: "c8d8a0d088e6eec5621699cb84fe085b"
server_key_version: "encryptionkey@0"
I20260923 19:29:46.307803 5229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:46.313724 5229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:46.314539 5229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:46.314778 5229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:46.315690 5229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:46.317093 5229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:46.317184 5229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:46.317296 5229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:46.317363 5229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:46.350935 5229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35483
I20260923 19:29:46.351015 5355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35483 every 8 connection(s)
I20260923 19:29:46.352218 5229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:46.356861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 5229
I20260923 19:29:46.357038 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:46.357456 388 external_mini_cluster.cc:1428] Setting key ff8a4972669a4e16a88de2d8040464d5
I20260923 19:29:46.360594 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45353
--builtin_ntp_servers=127.0.97.20:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 19:29:46.361418 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.355118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36979 (local address 127.0.97.62:45353)
0923 19:29:46.355301 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:46.355305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:46.356782 (+ 1477us) server_negotiation.cc:407] Connection header received
0923 19:29:46.358879 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:46.358884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:46.358942 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:46.359035 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:46.360584 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:46.360591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:46.360596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:46.360614 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:46.360649 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:46.360661 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:46.360769 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:46.361039 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 19:29:46.361121 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 19:29:46.362524 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.355551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45353 (local address 127.0.97.2:36979)
0923 19:29:46.356597 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:46.356638 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:46.358570 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:46.359331 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:46.359343 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:46.359476 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:46.360270 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:29:46.360272 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:46.360315 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:29:46.360408 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:46.360825 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:46.360836 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:46.360845 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 19:29:46.361117 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 19:29:46.361399 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":280,"threads_started":1}
I20260923 19:29:46.364131 5356 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45353
I20260923 19:29:46.364466 5356 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:46.365082 5356 heartbeater.cc:511] Master 127.0.97.62:45353 requested a full tablet report, sending...
I20260923 19:29:46.366771 5039 ts_manager.cc:194] Registered new tserver with Master: 576a5c665d7943b5951f9229175cc388 (127.0.97.2:35483)
W20260923 19:29:46.695868 5359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:46.696148 5359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:46.696256 5359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:46.703559 5359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:46.703667 5359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:46.703801 5359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:46.713059 5359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:46.714406 5359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:46.716121 5359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:46.729665 5364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:46.729632 5365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:46.729965 5367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:46.730783 5359 server_base.cc:1034] running on GCE node
I20260923 19:29:46.731436 5359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:46.732443 5359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:46.733700 5359 hybrid_clock.cc:648] HybridClock initialized: now 1790191786733638 us; error 49 us; skew 500 ppm
I20260923 19:29:46.734041 5359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:29:46.736575 5359 webserver.cc:460] Webserver started at http://127.0.97.3:36329/ using document root <none> and password file <none>
I20260923 19:29:46.737300 5359 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:46.737401 5359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:46.737695 5359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:46.740700 5359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "df3142b7bfdd45849de03027c95fc855"
format_stamp: "Formatted at 2026-09-23 19:29:46 on dist-test-slave-kxfb"
server_key: "b78d0978ce4da7a87daac4c715b37420"
server_key_iv: "bb595cd91063934a4648797dc13b5583"
server_key_version: "encryptionkey@0"
I20260923 19:29:46.741467 5359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "df3142b7bfdd45849de03027c95fc855"
format_stamp: "Formatted at 2026-09-23 19:29:46 on dist-test-slave-kxfb"
server_key: "b78d0978ce4da7a87daac4c715b37420"
server_key_iv: "bb595cd91063934a4648797dc13b5583"
server_key_version: "encryptionkey@0"
I20260923 19:29:46.746907 5359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 19:29:46.750499 5373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:46.752045 5359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:29:46.752218 5359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "df3142b7bfdd45849de03027c95fc855"
format_stamp: "Formatted at 2026-09-23 19:29:46 on dist-test-slave-kxfb"
server_key: "b78d0978ce4da7a87daac4c715b37420"
server_key_iv: "bb595cd91063934a4648797dc13b5583"
server_key_version: "encryptionkey@0"
I20260923 19:29:46.752442 5359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:46.757951 5359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:46.758737 5359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:46.758965 5359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:46.759868 5359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:46.761199 5359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:46.761291 5359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:46.761400 5359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:46.761467 5359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:46.793937 5359 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37527
I20260923 19:29:46.794026 5485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37527 every 8 connection(s)
I20260923 19:29:46.795275 5359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:46.802830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 5359
I20260923 19:29:46.803058 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:46.803583 388 external_mini_cluster.cc:1428] Setting key 9da72352e4678d825780eeed3f995e0a
I20260923 19:29:46.804708 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.798331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34909 (local address 127.0.97.62:45353)
0923 19:29:46.798543 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:46.798548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:46.800088 (+ 1540us) server_negotiation.cc:407] Connection header received
0923 19:29:46.801981 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:46.801985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:46.802045 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:46.802146 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:46.803804 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:46.803809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:46.803813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:46.803830 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:46.803865 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:46.803877 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:46.803988 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:46.804284 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 19:29:46.804368 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 19:29:46.805897 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.798829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45353 (local address 127.0.97.3:34909)
0923 19:29:46.799895 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:46.799939 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:46.801700 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:46.802448 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:46.802460 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:46.802605 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:46.803398 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:29:46.803401 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:46.803459 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:29:46.803586 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:46.804013 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:46.804023 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:46.804032 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 19:29:46.804358 (+ 326us) client_negotiation.cc:240] Negotiation successful
0923 19:29:46.804658 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":272,"threads_started":1}
I20260923 19:29:46.807430 5486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45353
I20260923 19:29:46.807749 5486 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:46.808396 5486 heartbeater.cc:511] Master 127.0.97.62:45353 requested a full tablet report, sending...
I20260923 19:29:46.809892 5039 ts_manager.cc:194] Registered new tserver with Master: df3142b7bfdd45849de03027c95fc855 (127.0.97.3:37527)
I20260923 19:29:46.819506 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:29:46.826587 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.824113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.0.97.62:45353)
0923 19:29:46.824332 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:46.824339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:46.824854 (+ 515us) server_negotiation.cc:407] Connection header received
0923 19:29:46.824997 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:46.825001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:46.825061 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:46.825146 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:46.825972 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:46.825976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:46.825980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:46.825991 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:46.826019 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:46.826030 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:46.826123 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:46.826193 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 19:29:46.826241 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 19:29:46.830757 5039 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:29:46.837958 5039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:29:46.871100 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.850935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45997 (local address 127.0.0.1:54434)
0923 19:29:46.866908 (+ 15973us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:46.866947 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:46.867117 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:46.867895 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:46.867903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:46.868014 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:46.869114 (+ 1100us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:29:46.869117 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:46.869148 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:29:46.869234 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:46.870315 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:46.870326 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:46.870339 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 19:29:46.870507 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 19:29:46.870605 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15696,"thread_start_us":15552,"threads_started":1}
I20260923 19:29:46.871552 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.851125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.0.97.1:45997)
0923 19:29:46.866406 (+ 15281us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:46.866415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:46.867245 (+ 830us) server_negotiation.cc:407] Connection header received
0923 19:29:46.867322 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:46.867330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:46.867670 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:46.868734 (+ 1064us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:46.870052 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:46.870060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:46.870076 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:46.870099 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:46.870153 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:46.870173 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:46.871082 (+ 909us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:46.871151 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 19:29:46.871255 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15092,"thread_start_us":14963,"threads_started":1}
I20260923 19:29:46.871989 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.851777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.0.97.2:35483)
0923 19:29:46.862219 (+ 10442us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:46.862231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:46.868483 (+ 6252us) server_negotiation.cc:407] Connection header received
0923 19:29:46.868908 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:46.868920 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:46.869215 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:46.869358 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:46.871035 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:46.871043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:46.871059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:46.871087 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:46.871150 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:46.871170 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:46.871302 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:46.871521 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 19:29:46.871636 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10256,"thread_start_us":10125,"threads_started":1}
I20260923 19:29:46.872854 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.851271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35483 (local address 127.0.0.1:54370)
0923 19:29:46.868296 (+ 17025us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:46.868320 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:46.868706 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:46.870115 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:46.870119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:46.870140 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:46.870823 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:29:46.870827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:46.870841 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:29:46.870879 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:46.871357 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:46.871362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:46.871368 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 19:29:46.872277 (+ 909us) client_negotiation.cc:240] Negotiation successful
0923 19:29:46.872336 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16846,"thread_start_us":16688,"threads_started":1}
I20260923 19:29:46.873088 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.851935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.0.97.3:37527)
0923 19:29:46.867723 (+ 15788us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:46.867735 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:46.869125 (+ 1390us) server_negotiation.cc:407] Connection header received
0923 19:29:46.869267 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:46.869280 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:46.869458 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:46.869607 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:46.872121 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:46.872130 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:46.872145 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:46.872163 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:46.872220 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:46.872238 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:46.872400 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:46.872636 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 19:29:46.872729 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15577,"thread_start_us":15293,"threads_started":1}
I20260923 19:29:46.874689 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:46.851736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37527 (local address 127.0.0.1:34752)
0923 19:29:46.868989 (+ 17253us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:46.869008 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:46.869130 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:46.869648 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:46.869654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:46.869675 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:46.870570 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:29:46.870573 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:46.870589 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:29:46.870626 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:46.872403 (+ 1777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:46.872408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:46.872413 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 19:29:46.872566 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 19:29:46.872626 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17129,"thread_start_us":16663,"threads_started":1}
I20260923 19:29:46.875731 5160 tablet_service.cc:1467] Processing CreateTablet for tablet cf274b285e9b47e581698ce653ba982e (DEFAULT_TABLE table=test-table [id=a702f35c670e444194d2c6815824eba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:46.877009 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf274b285e9b47e581698ce653ba982e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:46.876827 5291 tablet_service.cc:1467] Processing CreateTablet for tablet cf274b285e9b47e581698ce653ba982e (DEFAULT_TABLE table=test-table [id=a702f35c670e444194d2c6815824eba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:46.878432 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf274b285e9b47e581698ce653ba982e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:46.878175 5421 tablet_service.cc:1467] Processing CreateTablet for tablet cf274b285e9b47e581698ce653ba982e (DEFAULT_TABLE table=test-table [id=a702f35c670e444194d2c6815824eba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:29:46.879511 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf274b285e9b47e581698ce653ba982e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:46.888752 5504 tablet_bootstrap.cc:492] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933: Bootstrap starting.
I20260923 19:29:46.892202 5504 tablet_bootstrap.cc:654] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:46.893211 5504 log.cc:824] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:46.894121 5505 tablet_bootstrap.cc:492] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: Bootstrap starting.
I20260923 19:29:46.897913 5505 tablet_bootstrap.cc:654] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:46.898216 5504 tablet_bootstrap.cc:492] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933: No bootstrap required, opened a new log
I20260923 19:29:46.898517 5504 ts_tablet_manager.cc:1397] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260923 19:29:46.899102 5505 log.cc:824] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:46.899996 5506 tablet_bootstrap.cc:492] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: Bootstrap starting.
I20260923 19:29:46.902608 5505 tablet_bootstrap.cc:492] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: No bootstrap required, opened a new log
I20260923 19:29:46.902920 5505 ts_tablet_manager.cc:1397] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260923 19:29:46.903519 5506 tablet_bootstrap.cc:654] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:46.904852 5506 log.cc:824] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:46.910905 5504 raft_consensus.cc:348] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } }
I20260923 19:29:46.911837 5504 raft_consensus.cc:374] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:46.912009 5504 raft_consensus.cc:729] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe8774e5a58499da5c812945db88933, State: Initialized, Role: FOLLOWER
I20260923 19:29:46.913008 5504 consensus_queue.cc:260] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } }
I20260923 19:29:46.913480 5505 raft_consensus.cc:348] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } }
I20260923 19:29:46.914043 5505 raft_consensus.cc:374] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:46.914206 5505 raft_consensus.cc:729] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576a5c665d7943b5951f9229175cc388, State: Initialized, Role: FOLLOWER
I20260923 19:29:46.915112 5505 consensus_queue.cc:260] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } }
I20260923 19:29:46.916837 5356 heartbeater.cc:503] Master 127.0.97.62:45353 was elected leader, sending a full tablet report...
I20260923 19:29:46.918846 5505 ts_tablet_manager.cc:1428] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260923 19:29:46.923810 5225 heartbeater.cc:503] Master 127.0.97.62:45353 was elected leader, sending a full tablet report...
I20260923 19:29:46.924332 5506 tablet_bootstrap.cc:492] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: No bootstrap required, opened a new log
I20260923 19:29:46.924646 5506 ts_tablet_manager.cc:1397] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.009s
I20260923 19:29:46.925233 5504 ts_tablet_manager.cc:1428] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933: Time spent starting tablet: real 0.027s user 0.015s sys 0.007s
I20260923 19:29:46.932453 5506 raft_consensus.cc:348] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } }
I20260923 19:29:46.932802 5506 raft_consensus.cc:374] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:46.932888 5506 raft_consensus.cc:729] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3142b7bfdd45849de03027c95fc855, State: Initialized, Role: FOLLOWER
I20260923 19:29:46.933566 5506 consensus_queue.cc:260] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } }
I20260923 19:29:46.934808 5486 heartbeater.cc:503] Master 127.0.97.62:45353 was elected leader, sending a full tablet report...
I20260923 19:29:46.935227 5506 ts_tablet_manager.cc:1428] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
W20260923 19:29:47.047060 5487 tablet.cc:2369] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:29:47.104346 5357 tablet.cc:2369] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:29:47.140913 5511 raft_consensus.cc:482] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:29:47.141265 5511 raft_consensus.cc:504] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } }
I20260923 19:29:47.143106 5511 leader_election.cc:288] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fe8774e5a58499da5c812945db88933 (127.0.97.1:45997), df3142b7bfdd45849de03027c95fc855 (127.0.97.3:37527)
I20260923 19:29:47.147032 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.143748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54519 (local address 127.0.97.1:45997)
0923 19:29:47.143972 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:47.143978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:47.144762 (+ 784us) server_negotiation.cc:407] Connection header received
0923 19:29:47.144946 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:47.144953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:47.145033 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:47.145158 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:47.146328 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:47.146334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:47.146339 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:47.146360 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:47.146404 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:47.146420 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:47.146544 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:47.146657 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 19:29:47.146729 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260923 19:29:47.147032 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.143863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58791 (local address 127.0.97.3:37527)
0923 19:29:47.144020 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:47.144026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:47.144761 (+ 735us) server_negotiation.cc:407] Connection header received
0923 19:29:47.144946 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:47.144953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:47.145033 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:47.145158 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:47.146330 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:47.146335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:47.146340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:47.146360 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:47.146404 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:47.146420 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:47.146546 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:47.146657 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 19:29:47.146729 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 19:29:47.147271 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.143604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37527 (local address 127.0.97.2:58791)
0923 19:29:47.144626 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:47.144641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:47.144785 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:47.145205 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:47.145210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:47.145231 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:47.146063 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:29:47.146067 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:47.146090 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:29:47.146141 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:47.146527 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:47.146533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:47.146539 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 19:29:47.146715 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 19:29:47.146782 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":586,"threads_started":1}
I20260923 19:29:47.147645 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.143537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45997 (local address 127.0.97.2:54519)
0923 19:29:47.144592 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:47.144618 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:47.144794 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:47.145200 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:47.145205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:47.145231 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:47.146118 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:29:47.146121 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:47.146132 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:29:47.146159 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:47.146527 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:47.146533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:47.146539 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 19:29:47.146711 (+ 172us) client_negotiation.cc:240] Negotiation successful
0923 19:29:47.146795 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":20,"spinlock_wait_cycles":41856,"thread_start_us":653,"threads_started":1}
I20260923 19:29:47.148417 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf274b285e9b47e581698ce653ba982e" candidate_uuid: "576a5c665d7943b5951f9229175cc388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3142b7bfdd45849de03027c95fc855" is_pre_election: true
I20260923 19:29:47.148669 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf274b285e9b47e581698ce653ba982e" candidate_uuid: "576a5c665d7943b5951f9229175cc388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe8774e5a58499da5c812945db88933" is_pre_election: true
I20260923 19:29:47.148990 5441 raft_consensus.cc:2454] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 576a5c665d7943b5951f9229175cc388 in term 0.
I20260923 19:29:47.149178 5180 raft_consensus.cc:2454] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 576a5c665d7943b5951f9229175cc388 in term 0.
I20260923 19:29:47.149797 5245 leader_election.cc:302] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 576a5c665d7943b5951f9229175cc388, 5fe8774e5a58499da5c812945db88933, df3142b7bfdd45849de03027c95fc855; no voters:
I20260923 19:29:47.150396 5511 raft_consensus.cc:2781] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:29:47.150544 5511 raft_consensus.cc:482] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:29:47.150620 5511 raft_consensus.cc:3037] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:47.152782 5511 raft_consensus.cc:504] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } }
I20260923 19:29:47.153342 5511 leader_election.cc:288] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [CANDIDATE]: Term 1 election: Requested vote from peers 5fe8774e5a58499da5c812945db88933 (127.0.97.1:45997), df3142b7bfdd45849de03027c95fc855 (127.0.97.3:37527)
I20260923 19:29:47.153904 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf274b285e9b47e581698ce653ba982e" candidate_uuid: "576a5c665d7943b5951f9229175cc388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe8774e5a58499da5c812945db88933"
I20260923 19:29:47.153903 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf274b285e9b47e581698ce653ba982e" candidate_uuid: "576a5c665d7943b5951f9229175cc388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3142b7bfdd45849de03027c95fc855"
I20260923 19:29:47.154098 5180 raft_consensus.cc:3037] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:47.154098 5441 raft_consensus.cc:3037] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:47.157213 5180 raft_consensus.cc:2454] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 576a5c665d7943b5951f9229175cc388 in term 1.
I20260923 19:29:47.157213 5441 raft_consensus.cc:2454] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 576a5c665d7943b5951f9229175cc388 in term 1.
I20260923 19:29:47.157706 5244 leader_election.cc:302] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576a5c665d7943b5951f9229175cc388, 5fe8774e5a58499da5c812945db88933; no voters:
I20260923 19:29:47.158043 5511 raft_consensus.cc:2781] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:47.158593 5511 raft_consensus.cc:686] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 1 LEADER]: Becoming Leader. State: Replica: 576a5c665d7943b5951f9229175cc388, State: Running, Role: LEADER
I20260923 19:29:47.159152 5511 consensus_queue.cc:237] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } }
W20260923 19:29:47.161168 5226 tablet.cc:2369] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:29:47.164175 5039 catalog_manager.cc:5495] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 reported cstate change: term changed from 0 to 1, leader changed from <none> to 576a5c665d7943b5951f9229175cc388 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "576a5c665d7943b5951f9229175cc388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "576a5c665d7943b5951f9229175cc388" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 } health_report { overall_health: UNKNOWN } } }
I20260923 19:29:47.246374 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.244048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.0.97.2:35483)
0923 19:29:47.244295 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:47.244301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:47.244384 (+ 83us) server_negotiation.cc:407] Connection header received
0923 19:29:47.244596 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:47.244601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:47.244716 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:47.244820 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:47.245738 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:47.245742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:47.245747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:47.245762 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:47.245796 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:47.245809 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:47.245904 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:47.245968 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 19:29:47.246020 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 19:29:47.255427 5180 raft_consensus.cc:1261] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933 [term 1 FOLLOWER]: Refusing update from remote peer 576a5c665d7943b5951f9229175cc388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:47.255623 5441 raft_consensus.cc:1261] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [term 1 FOLLOWER]: Refusing update from remote peer 576a5c665d7943b5951f9229175cc388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:29:47.256369 5515 consensus_queue.cc:1035] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fe8774e5a58499da5c812945db88933" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:47.256861 5511 consensus_queue.cc:1035] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3142b7bfdd45849de03027c95fc855" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:47.270898 5518 mvcc.cc:205] Tried to move back new op lower bound from 7332625560588083200 to 7332625560208035840. Current Snapshot: MvccSnapshot[applied={T|T < 7332625560588083200}]
I20260923 19:29:47.284657 5039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:47.284976 5039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:29:47.289034 5039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c34020bbd3844e2a1e3da59bbdedfde: Sending DeleteTablet for 3 replicas of tablet cf274b285e9b47e581698ce653ba982e
I20260923 19:29:47.290479 5160 tablet_service.cc:1514] Processing DeleteTablet for tablet cf274b285e9b47e581698ce653ba982e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:47 UTC) from {username='slave'} at 127.0.0.1:54434
I20260923 19:29:47.290495 5291 tablet_service.cc:1514] Processing DeleteTablet for tablet cf274b285e9b47e581698ce653ba982e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:47 UTC) from {username='slave'} at 127.0.0.1:54370
I20260923 19:29:47.290652 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet cf274b285e9b47e581698ce653ba982e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:29:47 UTC) from {username='slave'} at 127.0.0.1:34752
I20260923 19:29:47.293437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 5098
I20260923 19:29:47.321935 5528 tablet_replica.cc:330] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933: stopping tablet replica
I20260923 19:29:47.323557 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.294860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.0.97.1:45997)
0923 19:29:47.295056 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:47.295064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:47.319767 (+ 24703us) server_negotiation.cc:407] Connection header received
0923 19:29:47.320395 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:47.320405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:47.320580 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:47.320670 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:47.322652 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:47.322659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:47.322666 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:47.322692 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:47.322738 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:47.322756 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:47.322876 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:47.323102 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 19:29:47.323186 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 19:29:47.324851 5220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:47.339619 5529 tablet_replica.cc:330] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: stopping tablet replica
I20260923 19:29:47.339848 5530 tablet_replica.cc:330] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: stopping tablet replica
I20260923 19:29:47.340394 5530 raft_consensus.cc:2229] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:47.341007 5530 raft_consensus.cc:2258] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:47.340974 5529 raft_consensus.cc:2229] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:29:47.342473 5529 raft_consensus.cc:2258] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:29:47.344036 5530 ts_tablet_manager.cc:1905] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:47.345722 5529 ts_tablet_manager.cc:1905] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:29:47.352306 5530 ts_tablet_manager.cc:1918] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:29:47.352649 5530 log.cc:1196] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/cf274b285e9b47e581698ce653ba982e
I20260923 19:29:47.353259 5530 ts_tablet_manager.cc:1939] T cf274b285e9b47e581698ce653ba982e P df3142b7bfdd45849de03027c95fc855: Deleting consensus metadata
I20260923 19:29:47.355886 5024 catalog_manager.cc:4841] TS df3142b7bfdd45849de03027c95fc855 (127.0.97.3:37527): tablet cf274b285e9b47e581698ce653ba982e (table test-table [id=a702f35c670e444194d2c6815824eba5]) successfully deleted
I20260923 19:29:47.363736 5529 ts_tablet_manager.cc:1918] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:29:47.364064 5529 log.cc:1196] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/cf274b285e9b47e581698ce653ba982e
I20260923 19:29:47.364774 5529 ts_tablet_manager.cc:1939] T cf274b285e9b47e581698ce653ba982e P 576a5c665d7943b5951f9229175cc388: Deleting consensus metadata
I20260923 19:29:47.367199 5025 catalog_manager.cc:4841] TS 576a5c665d7943b5951f9229175cc388 (127.0.97.2:35483): tablet cf274b285e9b47e581698ce653ba982e (table test-table [id=a702f35c670e444194d2c6815824eba5]) successfully deleted
I20260923 19:29:47.335103 5528 raft_consensus.cc:2229] T cf274b285e9b47e581698ce653ba982e P 5fe8774e5a58499da5c812945db88933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:29:47.444459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 5098
W20260923 19:29:47.460664 5023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45997 (error 108)
I20260923 19:29:47.461525 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 5229
I20260923 19:29:47.463321 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.462156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45997 (local address 127.0.0.1:54454)
0923 19:29:47.462993 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:47.463112 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":644,"threads_started":1}
W20260923 19:29:47.463600 5023 catalog_manager.cc:4584] TS 5fe8774e5a58499da5c812945db88933 (127.0.97.1:45997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf274b285e9b47e581698ce653ba982e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45997: connect: Connection refused (error 111)
I20260923 19:29:47.465077 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.462372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.0.97.2:35483)
0923 19:29:47.462608 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:47.462615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:47.462903 (+ 288us) server_negotiation.cc:407] Connection header received
0923 19:29:47.463007 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:47.463013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:47.463084 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:47.463190 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:47.464393 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:47.464397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:47.464401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:47.464417 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:47.464451 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:47.464463 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:47.464592 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:47.464708 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 19:29:47.464767 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 19:29:47.465813 5351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:47.517990 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.517429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45997 (local address 127.0.0.1:54466)
0923 19:29:47.517709 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:47.517835 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260923 19:29:47.519106 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.518754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45997 (local address 127.0.0.1:54472)
0923 19:29:47.518909 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:47.518986 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 19:29:47.580063 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.579214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45997 (local address 127.0.0.1:54476)
0923 19:29:47.579643 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:47.579768 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260923 19:29:47.581022 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 5229
I20260923 19:29:47.582985 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.580944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45997 (local address 127.0.0.1:54480)
0923 19:29:47.581116 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:47.581223 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 19:29:47.597774 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 5359
I20260923 19:29:47.600905 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.598676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.0.97.3:37527)
0923 19:29:47.599010 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:47.599016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:47.599037 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:29:47.599098 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:47.599103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:47.599188 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:47.599285 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:47.600218 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:47.600223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:47.600226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:47.600241 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:47.600310 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:47.600323 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:47.600437 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:47.600505 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 19:29:47.600562 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260923 19:29:47.601648 5481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:47.664173 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.663695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45997 (local address 127.0.0.1:54488)
0923 19:29:47.663954 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:47.664032 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 19:29:47.665642 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:47.665066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45997 (local address 127.0.0.1:54504)
0923 19:29:47.665312 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:47.665442 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260923 19:29:47.696200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 5359
I20260923 19:29:47.710220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 5008
I20260923 19:29:47.711699 5069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:47.815115 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 5008
2026-09-23T19:29:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3007 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 19:29:47.855896 388 mini_oidc.cc:152] Starting JWKS server
I20260923 19:29:47.858536 388 webserver.cc:460] Webserver started at https://127.0.0.1:38847/ using document root <none> and password file <none>
I20260923 19:29:47.858814 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 19:29:47.859479 388 webserver.cc:460] Webserver started at http://127.0.0.1:45573/ using document root <none> and password file <none>
2026-09-23T19:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:47Z Disabled control of system clock
I20260923 19:29:47.884601 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45681
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45681
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38847/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/testchainca.cert with env {}
W20260923 19:29:48.203549 5543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:48.203905 5543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:48.203995 5543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:48.211330 5543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:48.211453 5543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:48.211491 5543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:48.211537 5543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:48.211586 5543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 19:29:48.211650 5543 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 19:29:48.211695 5543 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38847/jwks.json
I20260923 19:29:48.221719 5543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38847/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:48.222976 5543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:48.224170 5543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:48.236763 5551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:48.236930 5549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:48.237646 5548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:48.237871 5543 server_base.cc:1034] running on GCE node
I20260923 19:29:48.238720 5543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:48.240150 5543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:48.241375 5543 hybrid_clock.cc:648] HybridClock initialized: now 1790191788241306 us; error 62 us; skew 500 ppm
I20260923 19:29:48.241739 5543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:29:48.244014 5543 webserver.cc:460] Webserver started at http://127.0.97.62:46377/ using document root <none> and password file <none>
I20260923 19:29:48.244678 5543 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:48.244765 5543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:48.245064 5543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:48.247884 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "89090232ff904b27871e7ea380484ad0"
format_stamp: "Formatted at 2026-09-23 19:29:48 on dist-test-slave-kxfb"
server_key: "a726f715ecf2417e5e0e59d68df2952f"
server_key_iv: "fb40bd6ec4d6a2181825e5a983cf0105"
server_key_version: "encryptionkey@0"
I20260923 19:29:48.248615 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "89090232ff904b27871e7ea380484ad0"
format_stamp: "Formatted at 2026-09-23 19:29:48 on dist-test-slave-kxfb"
server_key: "a726f715ecf2417e5e0e59d68df2952f"
server_key_iv: "fb40bd6ec4d6a2181825e5a983cf0105"
server_key_version: "encryptionkey@0"
I20260923 19:29:48.253767 5543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 19:29:48.257318 5557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:48.258862 5543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:48.259024 5543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "89090232ff904b27871e7ea380484ad0"
format_stamp: "Formatted at 2026-09-23 19:29:48 on dist-test-slave-kxfb"
server_key: "a726f715ecf2417e5e0e59d68df2952f"
server_key_iv: "fb40bd6ec4d6a2181825e5a983cf0105"
server_key_version: "encryptionkey@0"
I20260923 19:29:48.259202 5543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:48.278623 5543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:48.279433 5543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:48.279681 5543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:48.298810 5543 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45681
I20260923 19:29:48.298882 5608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45681 every 8 connection(s)
I20260923 19:29:48.300150 5543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:48.302066 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 5543
I20260923 19:29:48.302229 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:48.302906 388 external_mini_cluster.cc:1428] Setting key 8d0cdd3fc6d86b54742473fca7d8bf05
I20260923 19:29:48.305124 5609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:48.316330 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:48.304853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.0.97.62:45681)
0923 19:29:48.306022 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:48.306035 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:48.306085 (+ 50us) server_negotiation.cc:407] Connection header received
0923 19:29:48.307825 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:48.307847 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:48.308363 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:48.308747 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:48.309298 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:48.310685 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:48.311968 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:48.312635 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:48.312814 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:48.313927 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:48.313933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:48.313947 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:48.314562 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:48.314642 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:48.314667 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:48.314775 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:48.314995 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 19:29:48.315289 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":392,"threads_started":1}
I20260923 19:29:48.317595 5609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0: Bootstrap starting.
I20260923 19:29:48.320459 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:29:48.321352 5609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:48.322469 5609 log.cc:824] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:48.325459 5609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0: No bootstrap required, opened a new log
I20260923 19:29:48.332690 5609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89090232ff904b27871e7ea380484ad0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } }
I20260923 19:29:48.333048 5609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:48.333148 5609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89090232ff904b27871e7ea380484ad0, State: Initialized, Role: FOLLOWER
I20260923 19:29:48.333673 5609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89090232ff904b27871e7ea380484ad0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } }
I20260923 19:29:48.333889 5609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:48.334009 5609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:48.334162 5609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:48.336481 5609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89090232ff904b27871e7ea380484ad0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } }
I20260923 19:29:48.336957 5609 leader_election.cc:302] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89090232ff904b27871e7ea380484ad0; no voters:
I20260923 19:29:48.337534 5609 leader_election.cc:288] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:48.337733 5614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:48.338526 5614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [term 1 LEADER]: Becoming Leader. State: Replica: 89090232ff904b27871e7ea380484ad0, State: Running, Role: LEADER
I20260923 19:29:48.339282 5614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89090232ff904b27871e7ea380484ad0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } }
I20260923 19:29:48.339689 5609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:48.342744 5615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89090232ff904b27871e7ea380484ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89090232ff904b27871e7ea380484ad0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } } }
I20260923 19:29:48.342707 5616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89090232ff904b27871e7ea380484ad0. Latest consensus state: current_term: 1 leader_uuid: "89090232ff904b27871e7ea380484ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89090232ff904b27871e7ea380484ad0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } } }
I20260923 19:29:48.343210 5615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:48.343251 5616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:48.346782 5623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:48.351568 5623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:48.360817 5623 catalog_manager.cc:1347] Generated new cluster ID: 065a7df57da244459b5add80e7418b0e
I20260923 19:29:48.360944 5623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:48.381626 5623 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:48.383337 5623 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:48.393149 5623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89090232ff904b27871e7ea380484ad0: Generated new TSK 0
I20260923 19:29:48.394145 5623 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 19:29:48.436537 5639 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 19:29:48.436746 5639 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 19:29:48.437855 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:48.415278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.0.97.62:45681)
0923 19:29:48.415510 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:48.415517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:48.416114 (+ 597us) server_negotiation.cc:407] Connection header received
0923 19:29:48.416436 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:48.416443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:48.416543 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:48.416735 (+ 192us) server_negotiation.cc:226] Negotiated authn=JWT
0923 19:29:48.417306 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:48.418458 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:48.420050 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:48.420723 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:48.420856 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:48.421451 (+ 595us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 19:29:48.437144 (+ 15693us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 19:29:48.437522 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 19:29:48.437991 5612 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 19:29:48.439540 5638 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:29:48.415195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45681 (local address 127.0.0.1:47502)
0923 19:29:48.415996 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:48.416015 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:48.416199 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:48.416791 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:48.416796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:48.416809 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0923 19:29:48.417149 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:48.417158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:48.418673 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:48.418677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:48.419905 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:48.419917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:48.420865 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:48.420921 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:48.421064 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:48.421134 (+ 70us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 19:29:48.437467 (+ 16333us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 19:29:48.437821 (+ 354us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45681: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":431,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:45681: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38847/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 19:29:48.441802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 5543
I20260923 19:29:48.443120 5604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:48.535885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 5543
2026-09-23T19:29:48Z chronyd exiting
I20260923 19:29:48.586246 388 test_util.cc:175] -----------------------------------------------
I20260923 19:29:48.586408 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790191744267121-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (734 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 19:29:48.589989 388 mini_oidc.cc:152] Starting JWKS server
I20260923 19:29:48.591886 388 webserver.cc:460] Webserver started at https://127.0.0.1:35421/ using document root <none> and password file <none>
I20260923 19:29:48.592125 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 19:29:48.592798 388 webserver.cc:460] Webserver started at http://127.0.0.1:46491/ using document root <none> and password file <none>
2026-09-23T19:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:48Z Disabled control of system clock
I20260923 19:29:48.617811 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40367
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40367
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35421/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/testchainca.cert with env {}
W20260923 19:29:48.928938 5646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:48.929235 5646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:48.929343 5646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:48.936530 5646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:48.936645 5646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:48.936683 5646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:48.936765 5646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:48.936825 5646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 19:29:48.936918 5646 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 19:29:48.936966 5646 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35421/jwks.json
I20260923 19:29:48.946810 5646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35421/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:48.948165 5646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:48.949411 5646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:48.962409 5651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:48.962339 5654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:48.962399 5652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:48.963929 5646 server_base.cc:1034] running on GCE node
I20260923 19:29:48.964802 5646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:48.966396 5646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:48.967623 5646 hybrid_clock.cc:648] HybridClock initialized: now 1790191788967566 us; error 63 us; skew 500 ppm
I20260923 19:29:48.967988 5646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:29:48.974637 5646 webserver.cc:460] Webserver started at http://127.0.97.62:45129/ using document root <none> and password file <none>
I20260923 19:29:48.975342 5646 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:48.975450 5646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:48.975744 5646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:48.978535 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "04aaa44c1dd74372b5853c9676673d06"
format_stamp: "Formatted at 2026-09-23 19:29:48 on dist-test-slave-kxfb"
server_key: "330fdbd7538e0771679867435d6386e5"
server_key_iv: "0365348935b420df5acdfd6a0af9c249"
server_key_version: "encryptionkey@0"
I20260923 19:29:48.979270 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "04aaa44c1dd74372b5853c9676673d06"
format_stamp: "Formatted at 2026-09-23 19:29:48 on dist-test-slave-kxfb"
server_key: "330fdbd7538e0771679867435d6386e5"
server_key_iv: "0365348935b420df5acdfd6a0af9c249"
server_key_version: "encryptionkey@0"
I20260923 19:29:48.984477 5646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:29:48.988340 5660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:48.989951 5646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 19:29:48.990137 5646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "04aaa44c1dd74372b5853c9676673d06"
format_stamp: "Formatted at 2026-09-23 19:29:48 on dist-test-slave-kxfb"
server_key: "330fdbd7538e0771679867435d6386e5"
server_key_iv: "0365348935b420df5acdfd6a0af9c249"
server_key_version: "encryptionkey@0"
I20260923 19:29:48.990391 5646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:49.003631 5646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:49.004395 5646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:49.004637 5646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:49.022822 5646 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40367
I20260923 19:29:49.022893 5711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40367 every 8 connection(s)
I20260923 19:29:49.024091 5646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:49.028692 5712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:49.028882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 5646
I20260923 19:29:49.029115 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:49.029973 388 external_mini_cluster.cc:1428] Setting key 1925f1fd79a42d5b4db24d697749accf
I20260923 19:29:49.043073 5712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06: Bootstrap starting.
I20260923 19:29:49.044363 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:49.032300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.0.97.62:40367)
0923 19:29:49.033591 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:49.033612 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:49.033659 (+ 47us) server_negotiation.cc:407] Connection header received
0923 19:29:49.035910 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:49.035933 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:49.036444 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:49.036844 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:49.037390 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.038731 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.039965 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.040695 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.040886 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:49.041967 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:49.041975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:49.041990 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:49.042595 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:49.042680 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:49.042702 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:49.042812 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:49.043007 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 19:29:49.043342 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":557,"threads_started":1}
I20260923 19:29:49.047202 5712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:49.048363 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:29:49.048465 5712 log.cc:824] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:49.051779 5712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06: No bootstrap required, opened a new log
I20260923 19:29:49.059823 5712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa44c1dd74372b5853c9676673d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40367 } }
I20260923 19:29:49.060236 5712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:49.060336 5712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04aaa44c1dd74372b5853c9676673d06, State: Initialized, Role: FOLLOWER
I20260923 19:29:49.060992 5712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa44c1dd74372b5853c9676673d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40367 } }
I20260923 19:29:49.061233 5712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:49.061368 5712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:49.061543 5712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:49.063963 5712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa44c1dd74372b5853c9676673d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40367 } }
I20260923 19:29:49.064579 5712 leader_election.cc:302] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04aaa44c1dd74372b5853c9676673d06; no voters:
I20260923 19:29:49.065317 5712 leader_election.cc:288] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:49.065554 5717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:49.066380 5717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [term 1 LEADER]: Becoming Leader. State: Replica: 04aaa44c1dd74372b5853c9676673d06, State: Running, Role: LEADER
I20260923 19:29:49.067044 5717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa44c1dd74372b5853c9676673d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40367 } }
I20260923 19:29:49.067852 5712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:49.069819 5718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04aaa44c1dd74372b5853c9676673d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa44c1dd74372b5853c9676673d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40367 } } }
I20260923 19:29:49.070271 5718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:49.070163 5719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04aaa44c1dd74372b5853c9676673d06. Latest consensus state: current_term: 1 leader_uuid: "04aaa44c1dd74372b5853c9676673d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa44c1dd74372b5853c9676673d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40367 } } }
I20260923 19:29:49.070456 5719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:49.071265 5722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:49.076894 5722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:49.085515 5722 catalog_manager.cc:1347] Generated new cluster ID: 9b5b1e391aa64456bdc578fdd3330506
I20260923 19:29:49.085681 5722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:49.113298 5722 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:49.114794 5722 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:49.137871 5722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04aaa44c1dd74372b5853c9676673d06: Generated new TSK 0
I20260923 19:29:49.138787 5722 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 19:29:49.180258 5742 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 19:29:49.180438 5742 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 19:29:49.181403 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:49.155532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.0.97.62:40367)
0923 19:29:49.155744 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:49.155751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:49.156358 (+ 607us) server_negotiation.cc:407] Connection header received
0923 19:29:49.156547 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:49.156551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:49.156617 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:49.156744 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0923 19:29:49.157233 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.157977 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.159386 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.159863 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.159976 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:49.160471 (+ 495us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 19:29:49.180698 (+ 20227us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 19:29:49.181091 (+ 393us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34278: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 19:29:49.181555 5715 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34278: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 19:29:49.181890 5741 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:29:49.155365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40367 (local address 127.0.0.1:34278)
0923 19:29:49.156217 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:49.156236 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:49.156375 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:49.156787 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:49.156799 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:49.156811 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0923 19:29:49.157079 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:49.157088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.158116 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.158120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:49.159243 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:49.159260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.159977 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.160046 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:49.160191 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:49.160225 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 19:29:49.180951 (+ 20726us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 19:29:49.181215 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":583,"threads_started":1}
I20260923 19:29:49.183362 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 5646
I20260923 19:29:49.184819 5707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:49.287519 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 5646
2026-09-23T19:29:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 19:29:49.344919 388 mini_oidc.cc:152] Starting JWKS server
I20260923 19:29:49.347174 388 webserver.cc:460] Webserver started at https://127.0.0.1:42623/ using document root <none> and password file <none>
I20260923 19:29:49.347426 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 19:29:49.348176 388 webserver.cc:460] Webserver started at http://127.0.0.1:44625/ using document root <none> and password file <none>
2026-09-23T19:29:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:49Z Disabled control of system clock
I20260923 19:29:49.379110 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45091
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45091
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42623/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 19:29:49.693493 5749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:49.693800 5749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:49.693917 5749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:49.700937 5749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:49.701061 5749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:49.701114 5749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:49.701187 5749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:49.701244 5749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 19:29:49.701335 5749 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 19:29:49.701385 5749 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42623/jwks.json
I20260923 19:29:49.710494 5749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42623/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:49.711795 5749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:49.713455 5749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:49.725883 5757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:49.725970 5754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:49.725947 5755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:49.727254 5749 server_base.cc:1034] running on GCE node
I20260923 19:29:49.728010 5749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:49.729446 5749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:49.730654 5749 hybrid_clock.cc:648] HybridClock initialized: now 1790191789730582 us; error 67 us; skew 500 ppm
I20260923 19:29:49.731021 5749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:29:49.733361 5749 webserver.cc:460] Webserver started at http://127.0.97.62:38473/ using document root <none> and password file <none>
I20260923 19:29:49.734027 5749 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:49.734122 5749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:49.734464 5749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:49.737243 5749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "c230976480c944849fdc93101f0af32a"
format_stamp: "Formatted at 2026-09-23 19:29:49 on dist-test-slave-kxfb"
server_key: "289e63e79d767e546cb52066eb4aa315"
server_key_iv: "f0dc92a9a0997a89ee40cd6a62947c0b"
server_key_version: "encryptionkey@0"
I20260923 19:29:49.737974 5749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "c230976480c944849fdc93101f0af32a"
format_stamp: "Formatted at 2026-09-23 19:29:49 on dist-test-slave-kxfb"
server_key: "289e63e79d767e546cb52066eb4aa315"
server_key_iv: "f0dc92a9a0997a89ee40cd6a62947c0b"
server_key_version: "encryptionkey@0"
I20260923 19:29:49.743211 5749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 19:29:49.747007 5763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:49.748636 5749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:29:49.748816 5749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "c230976480c944849fdc93101f0af32a"
format_stamp: "Formatted at 2026-09-23 19:29:49 on dist-test-slave-kxfb"
server_key: "289e63e79d767e546cb52066eb4aa315"
server_key_iv: "f0dc92a9a0997a89ee40cd6a62947c0b"
server_key_version: "encryptionkey@0"
I20260923 19:29:49.749034 5749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:49.763159 5749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:49.763976 5749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:49.764242 5749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:49.783113 5749 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45091
I20260923 19:29:49.783186 5814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45091 every 8 connection(s)
I20260923 19:29:49.784495 5749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:49.789018 5815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:49.790887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 5749
I20260923 19:29:49.791087 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:49.791942 388 external_mini_cluster.cc:1428] Setting key 02b449cdb75c547e469f0a4cc160893f
I20260923 19:29:49.801900 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a: Bootstrap starting.
I20260923 19:29:49.805593 5815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:49.806566 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:49.794354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.0.97.62:45091)
0923 19:29:49.795627 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:49.795644 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:49.795693 (+ 49us) server_negotiation.cc:407] Connection header received
0923 19:29:49.797587 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:49.797614 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:49.798171 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:49.798642 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:49.799222 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.800581 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.801846 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.802685 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.802856 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:49.804023 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:49.804033 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:49.804053 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:49.804657 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:49.804760 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:49.804783 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:49.804884 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:49.805113 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 19:29:49.805439 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":412,"threads_started":1}
I20260923 19:29:49.806890 5815 log.cc:824] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:49.810046 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a: No bootstrap required, opened a new log
I20260923 19:29:49.810338 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:29:49.817541 5815 raft_consensus.cc:348] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230976480c944849fdc93101f0af32a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } }
I20260923 19:29:49.817929 5815 raft_consensus.cc:374] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:49.818017 5815 raft_consensus.cc:729] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c230976480c944849fdc93101f0af32a, State: Initialized, Role: FOLLOWER
I20260923 19:29:49.818670 5815 consensus_queue.cc:260] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230976480c944849fdc93101f0af32a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } }
I20260923 19:29:49.818897 5815 raft_consensus.cc:388] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:49.819024 5815 raft_consensus.cc:482] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:49.819185 5815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:49.821396 5815 raft_consensus.cc:504] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230976480c944849fdc93101f0af32a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } }
I20260923 19:29:49.821921 5815 leader_election.cc:302] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c230976480c944849fdc93101f0af32a; no voters:
I20260923 19:29:49.822674 5815 leader_election.cc:288] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:49.822851 5820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:49.823598 5820 raft_consensus.cc:686] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [term 1 LEADER]: Becoming Leader. State: Replica: c230976480c944849fdc93101f0af32a, State: Running, Role: LEADER
I20260923 19:29:49.824173 5820 consensus_queue.cc:237] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230976480c944849fdc93101f0af32a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } }
I20260923 19:29:49.824893 5815 sys_catalog.cc:564] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:49.827291 5822 sys_catalog.cc:455] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c230976480c944849fdc93101f0af32a. Latest consensus state: current_term: 1 leader_uuid: "c230976480c944849fdc93101f0af32a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230976480c944849fdc93101f0af32a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } } }
I20260923 19:29:49.827669 5822 sys_catalog.cc:458] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:49.828171 5821 sys_catalog.cc:455] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c230976480c944849fdc93101f0af32a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230976480c944849fdc93101f0af32a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45091 } } }
I20260923 19:29:49.828504 5821 sys_catalog.cc:458] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:49.828848 5826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:49.835340 5826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:49.846385 5826 catalog_manager.cc:1347] Generated new cluster ID: 74e57264088c4b61bf76dcaf3f7a5d22
I20260923 19:29:49.846505 5826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:49.871940 5826 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:49.873454 5826 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:49.881551 5826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c230976480c944849fdc93101f0af32a: Generated new TSK 0
I20260923 19:29:49.882583 5826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:29:49.918220 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:49.909295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38778 (local address 127.0.97.62:45091)
0923 19:29:49.909580 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:49.909589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:49.910019 (+ 430us) server_negotiation.cc:407] Connection header received
0923 19:29:49.910314 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:49.910319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:49.910427 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:49.910640 (+ 213us) server_negotiation.cc:226] Negotiated authn=JWT
0923 19:29:49.911160 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.912045 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.914041 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.914657 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.914772 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:49.915389 (+ 617us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 19:29:49.917530 (+ 2141us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 19:29:49.917874 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38778: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":85}
W20260923 19:29:49.918577 5844 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:29:49.909136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45091 (local address 127.0.0.1:38778)
0923 19:29:49.909884 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:49.909907 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:49.910081 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:49.910676 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:49.910681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:49.910695 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0923 19:29:49.910998 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:49.911010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.912182 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.912187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:49.913877 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:49.913892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:49.914803 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:49.914877 (+ 74us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:49.915058 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:49.915102 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 19:29:49.917785 (+ 2683us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 19:29:49.918086 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45091: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":418,"threads_started":1}
W20260923 19:29:49.918730 5818 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38778: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260923 19:29:49.919744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 5749
I20260923 19:29:49.921051 5810 generic_service.cc:196] Checking for leaks (request via RPC)
E20260923 19:29:49.931357 5845 webserver.cc:533] Webserver: sslize failed:
E20260923 19:29:49.931478 5845 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 19:29:50.013111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 5749
2026-09-23T19:29:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (721 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T19:29:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:50Z Disabled control of system clock
I20260923 19:29:50.094465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46559
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46559
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/testchainca.cert with env {}
W20260923 19:29:50.401959 5850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:50.402263 5850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:50.402438 5850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:50.410429 5850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:50.410552 5850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:50.410588 5850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:50.410645 5850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:50.410692 5850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 19:29:50.410744 5850 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 19:29:50.410786 5850 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 19:29:50.421010 5850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46559
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:50.422320 5850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:50.423518 5850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:50.435273 5858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:50.435499 5855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:50.435458 5856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:50.436501 5850 server_base.cc:1034] running on GCE node
I20260923 19:29:50.437189 5850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:50.438566 5850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:50.439781 5850 hybrid_clock.cc:648] HybridClock initialized: now 1790191790439718 us; error 59 us; skew 500 ppm
I20260923 19:29:50.440128 5850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:29:50.442440 5850 webserver.cc:460] Webserver started at http://127.0.97.62:39745/ using document root <none> and password file <none>
I20260923 19:29:50.443082 5850 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:50.443181 5850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:50.443477 5850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:50.446170 5850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "30f5ab55037f42b6bb2c9a2572a43c59"
format_stamp: "Formatted at 2026-09-23 19:29:50 on dist-test-slave-kxfb"
server_key: "b70f29bfbf02754dd2d7d4f8fe427732"
server_key_iv: "efee0b4eeff449cb29c4f142a44bf359"
server_key_version: "encryptionkey@0"
I20260923 19:29:50.446947 5850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "30f5ab55037f42b6bb2c9a2572a43c59"
format_stamp: "Formatted at 2026-09-23 19:29:50 on dist-test-slave-kxfb"
server_key: "b70f29bfbf02754dd2d7d4f8fe427732"
server_key_iv: "efee0b4eeff449cb29c4f142a44bf359"
server_key_version: "encryptionkey@0"
I20260923 19:29:50.452111 5850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 19:29:50.455708 5864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:50.457149 5850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 19:29:50.457317 5850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "30f5ab55037f42b6bb2c9a2572a43c59"
format_stamp: "Formatted at 2026-09-23 19:29:50 on dist-test-slave-kxfb"
server_key: "b70f29bfbf02754dd2d7d4f8fe427732"
server_key_iv: "efee0b4eeff449cb29c4f142a44bf359"
server_key_version: "encryptionkey@0"
I20260923 19:29:50.457535 5850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:50.499831 5850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:50.500645 5850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:50.500907 5850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:50.519512 5850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46559
I20260923 19:29:50.519567 5915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46559 every 8 connection(s)
I20260923 19:29:50.520797 5850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:50.522828 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 5850
I20260923 19:29:50.523044 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:50.523661 388 external_mini_cluster.cc:1428] Setting key 9d25039595285f67f8fdfed2d4685d18
I20260923 19:29:50.525803 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:50.540017 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:50.525668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.0.97.62:46559)
0923 19:29:50.526867 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:50.526885 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:50.526949 (+ 64us) server_negotiation.cc:407] Connection header received
0923 19:29:50.529163 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:50.529194 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:50.529888 (+ 694us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:50.530416 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:50.531021 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:50.533041 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:50.534349 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:50.535394 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:50.535592 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:50.536799 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:50.536808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:50.536827 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:29:50.537656 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:50.537759 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:29:50.537785 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:50.537927 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:50.538197 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 19:29:50.538616 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":368,"threads_started":1}
I20260923 19:29:50.541024 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59: Bootstrap starting.
I20260923 19:29:50.544611 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:29:50.544700 5916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:50.545930 5916 log.cc:824] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:50.549156 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59: No bootstrap required, opened a new log
I20260923 19:29:50.556628 5916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f5ab55037f42b6bb2c9a2572a43c59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46559 } }
I20260923 19:29:50.557034 5916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:50.557109 5916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f5ab55037f42b6bb2c9a2572a43c59, State: Initialized, Role: FOLLOWER
I20260923 19:29:50.557686 5916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f5ab55037f42b6bb2c9a2572a43c59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46559 } }
I20260923 19:29:50.557895 5916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:50.557997 5916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:50.558125 5916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:50.560413 5916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f5ab55037f42b6bb2c9a2572a43c59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46559 } }
I20260923 19:29:50.561009 5916 leader_election.cc:302] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f5ab55037f42b6bb2c9a2572a43c59; no voters:
I20260923 19:29:50.561734 5916 leader_election.cc:288] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:50.561908 5921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:50.562709 5921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [term 1 LEADER]: Becoming Leader. State: Replica: 30f5ab55037f42b6bb2c9a2572a43c59, State: Running, Role: LEADER
I20260923 19:29:50.563422 5921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f5ab55037f42b6bb2c9a2572a43c59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46559 } }
I20260923 19:29:50.564473 5916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:50.566955 5922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f5ab55037f42b6bb2c9a2572a43c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f5ab55037f42b6bb2c9a2572a43c59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46559 } } }
I20260923 19:29:50.567437 5922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:50.567927 5923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f5ab55037f42b6bb2c9a2572a43c59. Latest consensus state: current_term: 1 leader_uuid: "30f5ab55037f42b6bb2c9a2572a43c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f5ab55037f42b6bb2c9a2572a43c59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46559 } } }
I20260923 19:29:50.568389 5923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:50.569522 5927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:50.576238 5927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:50.586591 5927 catalog_manager.cc:1347] Generated new cluster ID: 11b3dae82ea0403ca9735352810c8cd0
I20260923 19:29:50.586745 5927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:50.625869 5927 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:50.627827 5927 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:50.639744 5927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f5ab55037f42b6bb2c9a2572a43c59: Generated new TSK 0
I20260923 19:29:50.640935 5927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:29:50.765378 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:50.681224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.0.97.62:46559)
0923 19:29:50.681524 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:50.681531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:50.681943 (+ 412us) server_negotiation.cc:407] Connection header received
0923 19:29:50.682250 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:50.682254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:50.682392 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:50.682555 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0923 19:29:50.683070 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:50.683901 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:50.685959 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:50.686591 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:50.686721 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:50.687310 (+ 589us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 19:29:50.764646 (+ 77336us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 19:29:50.765072 (+ 426us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46380: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":90}
W20260923 19:29:50.765534 5919 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46380: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 19:29:50.765825 5945 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:29:50.681061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46559 (local address 127.0.0.1:46380)
0923 19:29:50.681801 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:50.681824 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:50.682021 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:50.682597 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:50.682603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:50.682617 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0923 19:29:50.682908 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:50.682920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:50.684057 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:50.684063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:50.685790 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:50.685804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:50.686719 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:50.686775 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:50.686957 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:50.687003 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 19:29:50.764980 (+ 77977us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 19:29:50.765337 (+ 357us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:46559: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":395,"threads_started":1}
I20260923 19:29:50.767431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 5850
I20260923 19:29:50.768709 5911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:50.880913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 5850
2026-09-23T19:29:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (847 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (451 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (438 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:51 dist-test-slave-kxfb krb5kdc[5960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:51 dist-test-slave-kxfb krb5kdc[5960](info): set up 2 sockets
Sep 23 19:29:51 dist-test-slave-kxfb krb5kdc[5960](info): commencing operation
krb5kdc: starting...
W20260923 19:29:52.779109 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:52 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:29:53.002677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44011
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:53.364804 5973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:53.365157 5973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:53.365266 5973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:53.372860 5973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:53.372988 5973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:53.373049 5973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:53.373111 5973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:53.373167 5973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:53.383432 5973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:53.385027 5973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:53.386575 5973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:53.399262 5978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:53.399333 5981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:53.399566 5979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:53.400450 5973 server_base.cc:1034] running on GCE node
I20260923 19:29:53.401368 5973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:53.403134 5973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:53.404384 5973 hybrid_clock.cc:648] HybridClock initialized: now 1790191793404322 us; error 51 us; skew 500 ppm
I20260923 19:29:53.404798 5973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:53 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:53.408025 5973 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:29:53.409577 5973 webserver.cc:460] Webserver started at http://127.0.97.62:41335/ using document root <none> and password file <none>
I20260923 19:29:53.410351 5973 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:53.410467 5973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:53.410775 5973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:53.413851 5973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f"
format_stamp: "Formatted at 2026-09-23 19:29:53 on dist-test-slave-kxfb"
server_key: "272a769e2572c48e384e5d424382f8d3"
server_key_iv: "374d329e0cdc955086cab8d533e51aac"
server_key_version: "encryptionkey@0"
I20260923 19:29:53.414662 5973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f"
format_stamp: "Formatted at 2026-09-23 19:29:53 on dist-test-slave-kxfb"
server_key: "272a769e2572c48e384e5d424382f8d3"
server_key_iv: "374d329e0cdc955086cab8d533e51aac"
server_key_version: "encryptionkey@0"
I20260923 19:29:53.420277 5973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:29:53.423979 5988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:53.425643 5973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 19:29:53.425828 5973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f"
format_stamp: "Formatted at 2026-09-23 19:29:53 on dist-test-slave-kxfb"
server_key: "272a769e2572c48e384e5d424382f8d3"
server_key_iv: "374d329e0cdc955086cab8d533e51aac"
server_key_version: "encryptionkey@0"
I20260923 19:29:53.426021 5973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:53.443382 5973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:53.447417 5973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:53.447666 5973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:53.466300 5973 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44011
I20260923 19:29:53.466377 6039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44011 every 8 connection(s)
I20260923 19:29:53.467674 5973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:53.472108 6040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:53.475901 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 5973
I20260923 19:29:53.476164 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:53.476964 388 external_mini_cluster.cc:1428] Setting key 0d005cb40f58eea41264776869a8d2f9
I20260923 19:29:53.486474 6040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f: Bootstrap starting.
I20260923 19:29:53.490114 6040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:53.491398 6040 log.cc:824] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f: Log is configured to *not* fsync() on all Append() calls
Sep 23 19:29:53 dist-test-slave-kxfb krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:53.494815 6040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f: No bootstrap required, opened a new log
I20260923 19:29:53.500581 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:53.479678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.0.97.62:44011)
0923 19:29:53.480615 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:53.480635 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:53.480682 (+ 47us) server_negotiation.cc:407] Connection header received
0923 19:29:53.482588 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:53.482615 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:53.483099 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:53.483511 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:53.484927 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:53.486243 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:53.487508 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:53.488230 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:53.488403 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:53.492707 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:53.492720 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:53.492753 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:53.492810 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:53.495926 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:53.496739 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:53.496769 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:53.496776 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:53.496961 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:53.497427 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:53.497430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:53.497432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:53.498084 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:53.498391 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:53.498866 (+ 475us) server_negotiation.cc:299] Negotiation successful
0923 19:29:53.499295 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":415,"threads_started":1}
I20260923 19:29:53.504256 6040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260923 19:29:53.504655 6040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:53.504742 6040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1db8a3f38b40ec9a1c6ad42aa2d99f, State: Initialized, Role: FOLLOWER
I20260923 19:29:53.505293 6040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260923 19:29:53.505617 6040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:53.505772 6040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:53.505926 6040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:53.508234 6040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260923 19:29:53.508829 6040 leader_election.cc:302] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f1db8a3f38b40ec9a1c6ad42aa2d99f; no voters:
I20260923 19:29:53.509460 6040 leader_election.cc:288] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:53.509621 6045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:53.510438 6045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [term 1 LEADER]: Becoming Leader. State: Replica: 0f1db8a3f38b40ec9a1c6ad42aa2d99f, State: Running, Role: LEADER
I20260923 19:29:53.511060 6045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260923 19:29:53.511624 6040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:53.513929 6046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } } }
I20260923 19:29:53.514357 6046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:53.514900 6047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f1db8a3f38b40ec9a1c6ad42aa2d99f. Latest consensus state: current_term: 1 leader_uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f1db8a3f38b40ec9a1c6ad42aa2d99f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } } }
I20260923 19:29:53.515218 6047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:53.517654 6051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:53.523169 6051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:53.532909 6051 catalog_manager.cc:1347] Generated new cluster ID: 0922ba80b7e844c797e3c833162ef997
I20260923 19:29:53.533037 6051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:53.551931 6051 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:53.553457 6051 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:53.563436 6051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f1db8a3f38b40ec9a1c6ad42aa2d99f: Generated new TSK 0
I20260923 19:29:53.564296 6051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:29:53.630129 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:54.004169 6066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:54.004573 6066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:54.004746 6066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:54.012914 6066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:54.013074 6066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:54.013243 6066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:54.024552 6066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:54.026223 6066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:54.027751 6066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:54.043025 6071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:54.042933 6072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:54.042959 6074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:54.044634 6066 server_base.cc:1034] running on GCE node
I20260923 19:29:54.045410 6066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:54.046610 6066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:54.047830 6066 hybrid_clock.cc:648] HybridClock initialized: now 1790191794047766 us; error 54 us; skew 500 ppm
I20260923 19:29:54.048197 6066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:54 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:54.055766 6066 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:29:54.057684 6066 webserver.cc:460] Webserver started at http://127.0.97.1:40497/ using document root <none> and password file <none>
I20260923 19:29:54.058604 6066 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:54.058736 6066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:54.059073 6066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:54.062676 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "f6ff353d2d0f408a9c1d33432083f558"
format_stamp: "Formatted at 2026-09-23 19:29:54 on dist-test-slave-kxfb"
server_key: "86decfdc3bf1ff0565038e00a74ca780"
server_key_iv: "fa4d7c0157e8b261c3433c4cce7f1d5d"
server_key_version: "encryptionkey@0"
I20260923 19:29:54.063606 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f6ff353d2d0f408a9c1d33432083f558"
format_stamp: "Formatted at 2026-09-23 19:29:54 on dist-test-slave-kxfb"
server_key: "86decfdc3bf1ff0565038e00a74ca780"
server_key_iv: "fa4d7c0157e8b261c3433c4cce7f1d5d"
server_key_version: "encryptionkey@0"
I20260923 19:29:54.070051 6066 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 19:29:54.074867 6081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:54.076776 6066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:29:54.077016 6066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "f6ff353d2d0f408a9c1d33432083f558"
format_stamp: "Formatted at 2026-09-23 19:29:54 on dist-test-slave-kxfb"
server_key: "86decfdc3bf1ff0565038e00a74ca780"
server_key_iv: "fa4d7c0157e8b261c3433c4cce7f1d5d"
server_key_version: "encryptionkey@0"
I20260923 19:29:54.077262 6066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:54.104838 6066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:54.109639 6066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:54.110013 6066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:54.111196 6066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:54.112749 6066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:54.112826 6066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:54.112926 6066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:54.112972 6066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:54.147915 6066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39339
I20260923 19:29:54.147984 6193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39339 every 8 connection(s)
I20260923 19:29:54.149405 6066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:54.155690 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 6066
I20260923 19:29:54.155922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:54.156737 388 external_mini_cluster.cc:1428] Setting key acf4e5f611dbd52f4f29a42a8d668daa
Sep 23 19:29:54 dist-test-slave-kxfb krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:54.171958 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:54.152326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39273 (local address 127.0.97.62:44011)
0923 19:29:54.152793 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:54.152799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:54.154366 (+ 1567us) server_negotiation.cc:407] Connection header received
0923 19:29:54.156935 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:54.156940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:54.157031 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:54.157127 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:54.158886 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:54.159959 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:54.161544 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:54.162078 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:54.162224 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:54.166031 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:54.166038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:54.166043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:54.166086 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:54.168816 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:54.169677 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:54.169681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:54.169684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:54.169809 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:54.170230 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:54.170233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:54.170235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:54.170549 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:54.170678 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:54.171486 (+ 808us) server_negotiation.cc:299] Negotiation successful
0923 19:29:54.171672 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":252,"threads_started":1}
I20260923 19:29:54.173436 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:54.152925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44011 (local address 127.0.97.1:39273)
0923 19:29:54.154133 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:54.154200 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:54.156662 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:54.157401 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:54.157413 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:54.158018 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:54.158702 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:54.158724 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:54.160124 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:54.160128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:54.161403 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:54.161415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:54.162393 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:54.162400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:54.162606 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:54.163349 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:54.163371 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:54.165773 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:54.168969 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:54.168978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:54.168996 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:54.169559 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:54.169905 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:54.169909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:54.169912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:54.170125 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:54.170670 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:54.170678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:54.171101 (+ 423us) client_negotiation.cc:769] Sending connection context
0923 19:29:54.171400 (+ 299us) client_negotiation.cc:240] Negotiation successful
0923 19:29:54.171919 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":605,"threads_started":1}
I20260923 19:29:54.175226 6194 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44011
I20260923 19:29:54.175627 6194 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:54.176625 6194 heartbeater.cc:511] Master 127.0.97.62:44011 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 19:29:54.179308 6005 ts_manager.cc:194] Registered new tserver with Master: f6ff353d2d0f408a9c1d33432083f558 (127.0.97.1:39339)
I20260923 19:29:54.181013 6005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:29:54.209894 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:54.544647 6200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:54.544930 6200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:54.545058 6200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:54.552336 6200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:54.552474 6200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:54.552628 6200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:54.562341 6200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:54.563606 6200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:54.564920 6200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:54.578860 6205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:54.578984 6206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:54.578899 6208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:54.580261 6200 server_base.cc:1034] running on GCE node
I20260923 19:29:54.581099 6200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:54.582322 6200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:54.583549 6200 hybrid_clock.cc:648] HybridClock initialized: now 1790191794583457 us; error 94 us; skew 500 ppm
I20260923 19:29:54.583952 6200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:54 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:54.587411 6200 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:29:54.589068 6200 webserver.cc:460] Webserver started at http://127.0.97.2:34757/ using document root <none> and password file <none>
I20260923 19:29:54.589862 6200 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:54.589986 6200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:54.590329 6200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:54.593695 6200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "d39b41447689496ab8d577acf4140789"
format_stamp: "Formatted at 2026-09-23 19:29:54 on dist-test-slave-kxfb"
server_key: "55520dd04d6b37af954d07d5f8983f8d"
server_key_iv: "70031499cbb3b189df7e77d2b8b30639"
server_key_version: "encryptionkey@0"
I20260923 19:29:54.594571 6200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d39b41447689496ab8d577acf4140789"
format_stamp: "Formatted at 2026-09-23 19:29:54 on dist-test-slave-kxfb"
server_key: "55520dd04d6b37af954d07d5f8983f8d"
server_key_iv: "70031499cbb3b189df7e77d2b8b30639"
server_key_version: "encryptionkey@0"
I20260923 19:29:54.600437 6200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 19:29:54.604390 6215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:54.606109 6200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 19:29:54.606360 6200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "d39b41447689496ab8d577acf4140789"
format_stamp: "Formatted at 2026-09-23 19:29:54 on dist-test-slave-kxfb"
server_key: "55520dd04d6b37af954d07d5f8983f8d"
server_key_iv: "70031499cbb3b189df7e77d2b8b30639"
server_key_version: "encryptionkey@0"
I20260923 19:29:54.606596 6200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:54.621541 6200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:54.625902 6200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:54.626214 6200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:54.627322 6200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:54.628829 6200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:54.628906 6200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:54.629078 6200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:54.629137 6200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:54.666719 6200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39243
I20260923 19:29:54.666764 6327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39243 every 8 connection(s)
I20260923 19:29:54.668695 6200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:54.672703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 6200
I20260923 19:29:54.672930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:54.673486 388 external_mini_cluster.cc:1428] Setting key 7f7827fa67411d85bf672dffd2b215a7
Sep 23 19:29:54 dist-test-slave-kxfb krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:54.691599 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:54.672759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39489 (local address 127.0.97.62:44011)
0923 19:29:54.674623 (+ 1864us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:54.674630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:54.674654 (+ 24us) server_negotiation.cc:407] Connection header received
0923 19:29:54.676042 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:54.676047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:54.676128 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:54.676236 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:54.677981 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:54.678793 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:54.680407 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:54.680882 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:54.681008 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:54.685259 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:54.685265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:54.685270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:54.685309 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:54.688324 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:54.689081 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:54.689086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:54.689089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:54.689245 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:54.689645 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:54.689650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:54.689653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:54.690010 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:54.690165 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:54.690913 (+ 748us) server_negotiation.cc:299] Negotiation successful
0923 19:29:54.691181 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1725,"thread_start_us":1615,"threads_started":1}
I20260923 19:29:54.692359 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:54.672244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44011 (local address 127.0.97.2:39489)
0923 19:29:54.673281 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:54.673352 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:54.675722 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:54.676512 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:54.676522 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:54.677110 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:54.677812 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:54.677833 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:54.678931 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:54.678935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:54.680265 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:54.680278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:54.681115 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:54.681122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:54.681353 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:54.682116 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:54.682144 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:54.685001 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:54.688496 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:54.688504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:54.688517 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:54.688971 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:54.689350 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:54.689353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:54.689355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:54.689543 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:54.690132 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:54.690139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:54.690548 (+ 409us) client_negotiation.cc:769] Sending connection context
0923 19:29:54.690781 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 19:29:54.691251 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":386,"threads_started":1}
I20260923 19:29:54.693825 6328 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44011
I20260923 19:29:54.694128 6328 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:54.694919 6328 heartbeater.cc:511] Master 127.0.97.62:44011 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:29:54.696905 6005 ts_manager.cc:194] Registered new tserver with Master: d39b41447689496ab8d577acf4140789 (127.0.97.2:39243)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 19:29:54.697894 6005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:29:54.725512 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:55.037273 6334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:55.037595 6334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:55.037714 6334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:55.045073 6334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:55.045221 6334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:55.045377 6334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:55.055311 6334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:55.056602 6334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:55.057857 6334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:55.070999 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:55.070989 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:55.071090 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:55.072602 6334 server_base.cc:1034] running on GCE node
I20260923 19:29:55.073360 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:55.074541 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:55.075764 6334 hybrid_clock.cc:648] HybridClock initialized: now 1790191795075705 us; error 51 us; skew 500 ppm
I20260923 19:29:55.076135 6334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:55.079849 6334 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:29:55.081805 6334 webserver.cc:460] Webserver started at http://127.0.97.3:45623/ using document root <none> and password file <none>
I20260923 19:29:55.082911 6334 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:55.083053 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:55.083395 6334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:55.087181 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "d8ba8b6b4ecd4e9781aefc74b3130859"
format_stamp: "Formatted at 2026-09-23 19:29:55 on dist-test-slave-kxfb"
server_key: "1ded2a21d60147c23a1ee2c7c8ba5f78"
server_key_iv: "4776744f6573c0519a88922af8023f94"
server_key_version: "encryptionkey@0"
I20260923 19:29:55.088021 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d8ba8b6b4ecd4e9781aefc74b3130859"
format_stamp: "Formatted at 2026-09-23 19:29:55 on dist-test-slave-kxfb"
server_key: "1ded2a21d60147c23a1ee2c7c8ba5f78"
server_key_iv: "4776744f6573c0519a88922af8023f94"
server_key_version: "encryptionkey@0"
I20260923 19:29:55.093911 6334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 19:29:55.098196 6349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:55.099891 6334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:29:55.100072 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "d8ba8b6b4ecd4e9781aefc74b3130859"
format_stamp: "Formatted at 2026-09-23 19:29:55 on dist-test-slave-kxfb"
server_key: "1ded2a21d60147c23a1ee2c7c8ba5f78"
server_key_iv: "4776744f6573c0519a88922af8023f94"
server_key_version: "encryptionkey@0"
I20260923 19:29:55.100298 6334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:55.114166 6334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:55.118873 6334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:55.119170 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:55.120247 6334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:55.121697 6334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:55.121779 6334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:55.121903 6334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:55.121961 6334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:55.156450 6334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43155
I20260923 19:29:55.156525 6461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43155 every 8 connection(s)
I20260923 19:29:55.158437 6334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:55.167508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 6334
I20260923 19:29:55.167722 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:55.168180 388 external_mini_cluster.cc:1428] Setting key 37c7000bfc2b6de81034c8ede2907552
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:29:55.180287 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:55.161334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36587 (local address 127.0.97.62:44011)
0923 19:29:55.161590 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.161598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.162881 (+ 1283us) server_negotiation.cc:407] Connection header received
0923 19:29:55.164809 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.164814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.164905 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.165014 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.166724 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.167670 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.169527 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.169985 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.170083 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:55.174130 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:55.174136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:55.174140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:55.174175 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:55.176872 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.177852 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.177857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.177860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.177995 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.178551 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.178555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.178558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.178903 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:55.179031 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:55.179816 (+ 785us) server_negotiation.cc:299] Negotiation successful
0923 19:29:55.180022 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 19:29:55.181195 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:55.161763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44011 (local address 127.0.97.3:36587)
0923 19:29:55.162678 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:55.162733 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:55.164521 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:55.165260 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:55.165269 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:55.165833 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:55.166548 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:55.166567 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.167829 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.167835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:55.169369 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:55.169382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.170384 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.170389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:55.170607 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:55.171352 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:55.171374 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:55.173880 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:55.177038 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:55.177046 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:55.177060 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:55.177679 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:55.178127 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:55.178131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:55.178134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:55.178380 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:55.179036 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:55.179042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:55.179446 (+ 404us) client_negotiation.cc:769] Sending connection context
0923 19:29:55.179686 (+ 240us) client_negotiation.cc:240] Negotiation successful
0923 19:29:55.180129 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":345,"threads_started":1}
I20260923 19:29:55.183099 6462 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44011
I20260923 19:29:55.183437 6462 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:55.184160 6462 heartbeater.cc:511] Master 127.0.97.62:44011 requested a full tablet report, sending...
I20260923 19:29:55.184744 6194 heartbeater.cc:503] Master 127.0.97.62:44011 was elected leader, sending a full tablet report...
I20260923 19:29:55.185679 6005 ts_manager.cc:194] Registered new tserver with Master: d8ba8b6b4ecd4e9781aefc74b3130859 (127.0.97.3:43155)
I20260923 19:29:55.186794 6005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36587
I20260923 19:29:55.195963 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/test-admin.keytab.
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:55.224473 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 19:29:55.230015 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5cc' of type FILE
W20260923 19:29:55.230065 6469 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:29:55.227297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.0.97.1:39339)
0923 19:29:55.228090 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.228100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.228136 (+ 36us) server_negotiation.cc:407] Connection header received
0923 19:29:55.228217 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.228225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.228494 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.228673 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.229583 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43784: BlockingRecv error: recv got EOF from 127.0.0.1:43784 (error 108)
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":399,"threads_started":1}
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:55.231673 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:55.232957 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:55.249042 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:55.234638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.0.97.1:39339)
0923 19:29:55.234884 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.234890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.235101 (+ 211us) server_negotiation.cc:407] Connection header received
0923 19:29:55.235246 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.235250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.235320 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.235413 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.236355 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.237391 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.238742 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.239382 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.239489 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:55.243179 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:55.243194 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:55.243213 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:55.243266 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:55.246213 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.246934 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.246942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.246944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.247075 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.247422 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.247430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.247432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.247858 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:55.248116 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:55.248379 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 19:29:55.248642 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260923 19:29:55.255160 6469 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:29:55.253446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.0.97.1:39339)
0923 19:29:55.253627 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.253631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.253883 (+ 252us) server_negotiation.cc:407] Connection header received
0923 19:29:55.254077 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.254082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.254147 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.254248 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.254929 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43804: BlockingRecv error: recv got EOF from 127.0.0.1:43804 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260923 19:29:55.255555 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5cc' of type FILE
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:55.257057 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:55.258493 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:55.272876 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:55.259886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.0.97.1:39339)
0923 19:29:55.260094 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.260100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.260426 (+ 326us) server_negotiation.cc:407] Connection header received
0923 19:29:55.260574 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.260578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.260644 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.260727 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.261602 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.262323 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.263480 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.263903 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.264020 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:55.267797 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:55.267803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:55.267806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:55.267854 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:55.270417 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.271139 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.271143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.271145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.271257 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.271551 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.271554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.271556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.271816 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:55.272004 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:55.272465 (+ 461us) server_negotiation.cc:299] Negotiation successful
0923 19:29:55.272632 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260923 19:29:55.277554 6469 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:29:55.275821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.0.97.1:39339)
0923 19:29:55.275960 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.275966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.276340 (+ 374us) server_negotiation.cc:407] Connection header received
0923 19:29:55.276493 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.276498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.276563 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.276636 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.277337 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43818: BlockingRecv error: recv got EOF from 127.0.0.1:43818 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 19:29:55.277944 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790191744267121-388-0/krb5kdc/krb5cc' of type FILE
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:55.279464 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:55.280803 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:29:55.294198 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:55.282081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.0.97.1:39339)
0923 19:29:55.282256 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.282261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.282600 (+ 339us) server_negotiation.cc:407] Connection header received
0923 19:29:55.282745 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.282749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.282800 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.282874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.283713 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.284413 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.285578 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.285982 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.286084 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:55.289409 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:55.289413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:55.289417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:55.289447 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:55.291861 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.292455 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.292460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.292462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.292587 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.292902 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.292907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.292910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.293180 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:55.293468 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:55.293700 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 19:29:55.293898 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 19:29:55.296592 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 6066
I20260923 19:29:55.309128 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:55.297406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.0.97.1:39339)
0923 19:29:55.297555 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.297559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.297972 (+ 413us) server_negotiation.cc:407] Connection header received
0923 19:29:55.298127 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.298130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.298184 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.298255 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.299261 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.299937 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.301248 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.301720 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.301831 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:55.304018 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:55.304029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:55.304033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:55.304065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:55.306595 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.307302 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.307306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.307308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.307481 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.307919 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.307923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.307925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.308173 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:55.308419 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:55.308694 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 19:29:55.308870 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 19:29:55.309742 6189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:55.394773 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 6066
I20260923 19:29:55.408955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 6200
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:29:55.424639 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:55.409875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.0.97.2:39243)
0923 19:29:55.410512 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.410520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.410549 (+ 29us) server_negotiation.cc:407] Connection header received
0923 19:29:55.410611 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.410617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.410839 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.410992 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.412001 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.413106 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.414332 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.414921 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.415034 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:55.419115 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:55.419125 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:55.419145 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:55.419177 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:55.421894 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.422655 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.422661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.422664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.422785 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.423131 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.423135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.423137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.423473 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:55.423755 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:55.424072 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 19:29:55.424327 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":286,"threads_started":1}
I20260923 19:29:55.426278 6323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:55.517562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 6200
I20260923 19:29:55.530568 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 6334
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:29:55.547395 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:55.531604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.0.97.3:43155)
0923 19:29:55.532389 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:55.532397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:55.532433 (+ 36us) server_negotiation.cc:407] Connection header received
0923 19:29:55.532510 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:55.532517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:55.532790 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:55.532955 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:55.533947 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.535113 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.536402 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:55.537043 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:55.537172 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:55.541336 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:55.541387 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:55.541409 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:55.541449 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:55.544326 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.545183 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.545190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.545192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.545329 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:55.545792 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:55.545796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:55.545798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:55.546166 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:55.546505 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:55.546843 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 19:29:55.547055 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":409,"threads_started":1}
I20260923 19:29:55.548879 6457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:55.638464 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 6334
I20260923 19:29:55.651384 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 5973
I20260923 19:29:55.652760 6035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:29:55.753422 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 5973
2026-09-23T19:29:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3990 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[6489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[6489](info): set up 2 sockets
Sep 23 19:29:55 dist-test-slave-kxfb krb5kdc[6489](info): commencing operation
krb5kdc: starting...
W20260923 19:29:56.767028 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:29:56 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:29:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:29:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260923 19:29:56.994861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37425
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:57.357136 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:57.357506 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:57.357630 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:57.365520 6502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:57.365666 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:57.365743 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:57.365828 6502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:57.365892 6502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:57.376780 6502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:57.378753 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:57.380241 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:57.393321 6510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:57.393291 6508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:57.393643 6507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:57.394328 6502 server_base.cc:1034] running on GCE node
I20260923 19:29:57.395174 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:57.396762 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:57.397985 6502 hybrid_clock.cc:648] HybridClock initialized: now 1790191797397914 us; error 50 us; skew 500 ppm
I20260923 19:29:57.398422 6502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:57 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191797, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:57.402091 6502 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260923 19:29:57.403952 6502 webserver.cc:460] Webserver started at http://127.0.97.62:37281/ using document root <none> and password file <none>
I20260923 19:29:57.404808 6502 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:57.404928 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:57.405278 6502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:57.408849 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "164e149b0070471e81a429e1c5fe1947"
format_stamp: "Formatted at 2026-09-23 19:29:57 on dist-test-slave-kxfb"
server_key: "c45b65e921ad01fca51ed164c9b202ee"
server_key_iv: "64c96c2504e0b45c021d9aa5726adbce"
server_key_version: "encryptionkey@0"
I20260923 19:29:57.409725 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "164e149b0070471e81a429e1c5fe1947"
format_stamp: "Formatted at 2026-09-23 19:29:57 on dist-test-slave-kxfb"
server_key: "c45b65e921ad01fca51ed164c9b202ee"
server_key_iv: "64c96c2504e0b45c021d9aa5726adbce"
server_key_version: "encryptionkey@0"
I20260923 19:29:57.415752 6502 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260923 19:29:57.419826 6517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:57.421540 6502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:29:57.421782 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "164e149b0070471e81a429e1c5fe1947"
format_stamp: "Formatted at 2026-09-23 19:29:57 on dist-test-slave-kxfb"
server_key: "c45b65e921ad01fca51ed164c9b202ee"
server_key_iv: "64c96c2504e0b45c021d9aa5726adbce"
server_key_version: "encryptionkey@0"
I20260923 19:29:57.421981 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:57.449354 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:57.454097 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:57.454411 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:57.477499 6502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37425
I20260923 19:29:57.477542 6568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37425 every 8 connection(s)
I20260923 19:29:57.479063 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:29:57.483142 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:57.486604 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 6502
I20260923 19:29:57.486833 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:29:57.487598 388 external_mini_cluster.cc:1428] Setting key ee714fc30b872bd68f34fb4ee39828c4
I20260923 19:29:57.494899 6569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: Bootstrap starting.
I20260923 19:29:57.498144 6569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:57.499171 6569 log.cc:824] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:57.502106 6569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: No bootstrap required, opened a new log
Sep 23 19:29:57 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:29:57.511093 6569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } }
I20260923 19:29:57.511547 6569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:57.511660 6569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164e149b0070471e81a429e1c5fe1947, State: Initialized, Role: FOLLOWER
I20260923 19:29:57.512229 6569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } }
I20260923 19:29:57.512454 6569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:29:57.512586 6569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:29:57.512753 6569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:57.515173 6569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } }
I20260923 19:29:57.515283 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:57.489966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.0.97.62:37425)
0923 19:29:57.491431 (+ 1465us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:57.491451 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:57.491503 (+ 52us) server_negotiation.cc:407] Connection header received
0923 19:29:57.493594 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:57.493623 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:57.494175 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:57.494674 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:57.496284 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:57.497769 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:57.499124 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:57.499950 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:57.500144 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:57.505066 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:57.505079 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:57.505110 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:57.505170 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:57.509680 (+ 4510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:57.510777 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:57.510786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:57.510794 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:57.511020 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:57.511530 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:57.511541 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:57.511545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:57.512409 (+ 864us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:57.512862 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:57.513374 (+ 512us) server_negotiation.cc:299] Negotiation successful
0923 19:29:57.513937 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":871,"threads_started":1}
I20260923 19:29:57.515751 6569 leader_election.cc:302] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 164e149b0070471e81a429e1c5fe1947; no voters:
I20260923 19:29:57.516484 6569 leader_election.cc:288] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:29:57.516661 6574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:29:57.518635 6569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:29:57.523583 6574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 1 LEADER]: Becoming Leader. State: Replica: 164e149b0070471e81a429e1c5fe1947, State: Running, Role: LEADER
I20260923 19:29:57.524250 6574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } }
I20260923 19:29:57.527655 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164e149b0070471e81a429e1c5fe1947. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } }
I20260923 19:29:57.528069 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:57.528478 6575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } }
I20260923 19:29:57.528901 6575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
W20260923 19:29:57.535683 6588 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:29:57.535811 6588 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:29:57.535910 6589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:29:57.537660 6589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:29:57.547324 6589 catalog_manager.cc:1347] Generated new cluster ID: 7cf1765f85b34a0d83e9aacd359ae234
I20260923 19:29:57.547462 6589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:29:57.585557 6589 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:29:57.587232 6589 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:29:57.595450 6589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: Generated new TSK 0
I20260923 19:29:57.596402 6589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260923 19:29:57.658465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37425
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:58.020743 6595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:58.021085 6595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:58.021230 6595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:58.029122 6595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:58.029279 6595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:58.029446 6595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:29:58.040303 6595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:58.041990 6595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:58.043560 6595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:58.058718 6600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:58.058847 6601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:58.059510 6603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:58.060019 6595 server_base.cc:1034] running on GCE node
I20260923 19:29:58.060792 6595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:58.061923 6595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:58.063184 6595 hybrid_clock.cc:648] HybridClock initialized: now 1790191798063101 us; error 71 us; skew 500 ppm
I20260923 19:29:58.063683 6595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:58 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191798, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:58.067261 6595 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 19:29:58.069046 6595 webserver.cc:460] Webserver started at http://127.0.97.1:34839/ using document root <none> and password file <none>
I20260923 19:29:58.069829 6595 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:58.069954 6595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:58.070253 6595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:58.073360 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "066538205b95484ca562199aa7740b8a"
format_stamp: "Formatted at 2026-09-23 19:29:58 on dist-test-slave-kxfb"
server_key: "5abeaf0b462df250c7d9c0d54bf5cc8b"
server_key_iv: "1d6ed920b662b3e8acc97c57e37e1957"
server_key_version: "encryptionkey@0"
I20260923 19:29:58.074178 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "066538205b95484ca562199aa7740b8a"
format_stamp: "Formatted at 2026-09-23 19:29:58 on dist-test-slave-kxfb"
server_key: "5abeaf0b462df250c7d9c0d54bf5cc8b"
server_key_iv: "1d6ed920b662b3e8acc97c57e37e1957"
server_key_version: "encryptionkey@0"
I20260923 19:29:58.080106 6595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 19:29:58.083861 6610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:58.085563 6595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:29:58.085752 6595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "066538205b95484ca562199aa7740b8a"
format_stamp: "Formatted at 2026-09-23 19:29:58 on dist-test-slave-kxfb"
server_key: "5abeaf0b462df250c7d9c0d54bf5cc8b"
server_key_iv: "1d6ed920b662b3e8acc97c57e37e1957"
server_key_version: "encryptionkey@0"
I20260923 19:29:58.085951 6595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:58.121155 6595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:58.125543 6595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:58.125814 6595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:58.126960 6595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:58.128392 6595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:58.128471 6595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:58.128604 6595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:58.128656 6595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:58.161772 6595 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43027
I20260923 19:29:58.161888 6722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43027 every 8 connection(s)
I20260923 19:29:58.163179 6595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:29:58.173338 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 6595
I20260923 19:29:58.173594 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:29:58.174325 388 external_mini_cluster.cc:1428] Setting key 709485216c07d87aedf3eaff61dfe6a1
Sep 23 19:29:58 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191798, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:29:58.185036 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:58.166319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33947 (local address 127.0.97.62:37425)
0923 19:29:58.166838 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:58.166843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:58.167841 (+ 998us) server_negotiation.cc:407] Connection header received
0923 19:29:58.170473 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:58.170479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:58.170571 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:58.170697 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:58.172505 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:58.173525 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:58.175556 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:58.176069 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:58.176200 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:58.179759 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:58.179764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:58.179768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:58.179802 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:58.182124 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:58.182831 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:58.182835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:58.182837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:58.182942 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:58.183353 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:58.183356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:58.183358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:58.183623 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:58.183747 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:58.184458 (+ 711us) server_negotiation.cc:299] Negotiation successful
0923 19:29:58.184689 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":255,"threads_started":1}
I20260923 19:29:58.186012 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:58.166836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:33947)
0923 19:29:58.167643 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:58.167693 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:58.170090 (+ 2397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:58.170977 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:58.170987 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:58.171655 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:58.172326 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:58.172344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:58.173734 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:58.173743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:58.175323 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:58.175341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:58.176331 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:58.176339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:58.176645 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:58.177286 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:58.177313 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:58.179511 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:58.182254 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:58.182263 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:58.182279 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:58.182707 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:58.183045 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:58.183050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:58.183054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:58.183242 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:58.183739 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:58.183747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:58.184076 (+ 329us) client_negotiation.cc:769] Sending connection context
0923 19:29:58.184320 (+ 244us) client_negotiation.cc:240] Negotiation successful
0923 19:29:58.184850 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":308,"threads_started":1}
I20260923 19:29:58.187742 6723 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37425
I20260923 19:29:58.188215 6723 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:58.189311 6723 heartbeater.cc:511] Master 127.0.97.62:37425 requested a full tablet report, sending...
I20260923 19:29:58.192235 6534 ts_manager.cc:194] Registered new tserver with Master: 066538205b95484ca562199aa7740b8a (127.0.97.1:43027)
I20260923 19:29:58.194554 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33947
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260923 19:29:58.227732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37425
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:58.588653 6729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:58.589001 6729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:58.589138 6729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:58.596887 6729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:58.597024 6729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:58.597148 6729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:29:58.607760 6729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:58.609397 6729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:58.611203 6729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:58.625820 6734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:58.625777 6735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:58.625854 6737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:58.626967 6729 server_base.cc:1034] running on GCE node
I20260923 19:29:58.627681 6729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:58.628780 6729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:58.630008 6729 hybrid_clock.cc:648] HybridClock initialized: now 1790191798629944 us; error 57 us; skew 500 ppm
I20260923 19:29:58.630401 6729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:58 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191798, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:58.633737 6729 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 19:29:58.635421 6729 webserver.cc:460] Webserver started at http://127.0.97.2:41169/ using document root <none> and password file <none>
I20260923 19:29:58.636207 6729 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:58.636329 6729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:58.636646 6729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:58.639930 6729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "aaad02bea99c40c8a91cbd47e320579a"
format_stamp: "Formatted at 2026-09-23 19:29:58 on dist-test-slave-kxfb"
server_key: "85573eb76ce8515321174f692b043a77"
server_key_iv: "0562ae69270fce6bed76e14c8db44d41"
server_key_version: "encryptionkey@0"
I20260923 19:29:58.640758 6729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "aaad02bea99c40c8a91cbd47e320579a"
format_stamp: "Formatted at 2026-09-23 19:29:58 on dist-test-slave-kxfb"
server_key: "85573eb76ce8515321174f692b043a77"
server_key_iv: "0562ae69270fce6bed76e14c8db44d41"
server_key_version: "encryptionkey@0"
I20260923 19:29:58.646589 6729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 19:29:58.650637 6744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:58.652450 6729 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:29:58.652663 6729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "aaad02bea99c40c8a91cbd47e320579a"
format_stamp: "Formatted at 2026-09-23 19:29:58 on dist-test-slave-kxfb"
server_key: "85573eb76ce8515321174f692b043a77"
server_key_iv: "0562ae69270fce6bed76e14c8db44d41"
server_key_version: "encryptionkey@0"
I20260923 19:29:58.652891 6729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:58.671016 6729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:58.675225 6729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:58.675488 6729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:58.676487 6729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:58.677832 6729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:58.677907 6729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:58.678014 6729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:58.678082 6729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:58.710073 6729 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37959
I20260923 19:29:58.710203 6856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37959 every 8 connection(s)
I20260923 19:29:58.711436 6729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:29:58.720441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 6729
I20260923 19:29:58.720656 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:29:58.721293 388 external_mini_cluster.cc:1428] Setting key af7d149d46c27b790b3d6543012e105d
Sep 23 19:29:58 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191798, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:29:58.734966 6859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:58.714506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45403 (local address 127.0.97.62:37425)
0923 19:29:58.715004 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:58.715009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:58.716215 (+ 1206us) server_negotiation.cc:407] Connection header received
0923 19:29:58.718363 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:58.718374 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:58.718455 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:58.718578 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:58.720512 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:58.721588 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:58.723626 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:58.724270 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:58.724427 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:58.728631 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:58.728639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:58.728644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:58.728690 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:58.731444 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:58.732358 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:58.732364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:58.732368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:58.732528 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:58.733041 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:58.733045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:58.733048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:58.733385 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:58.733539 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:58.734425 (+ 886us) server_negotiation.cc:299] Negotiation successful
0923 19:29:58.734632 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260923 19:29:58.736152 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:58.715091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.2:45403)
0923 19:29:58.715977 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:58.716030 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:58.717995 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:58.718916 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:58.718927 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:58.719555 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:58.720289 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:58.720308 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:58.721816 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:58.721822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:58.723482 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:58.723495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:58.724507 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:58.724512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:58.724756 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:58.725527 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:58.725552 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:58.728317 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:58.731623 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:58.731632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:58.731666 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:58.732176 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:58.732692 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:58.732697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:58.732700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:58.732910 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:58.733533 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:58.733540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:58.733987 (+ 447us) client_negotiation.cc:769] Sending connection context
0923 19:29:58.734260 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 19:29:58.734796 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":347,"threads_started":1}
I20260923 19:29:58.737874 6857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37425
I20260923 19:29:58.738219 6857 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:58.739085 6857 heartbeater.cc:511] Master 127.0.97.62:37425 requested a full tablet report, sending...
I20260923 19:29:58.740763 6534 ts_manager.cc:194] Registered new tserver with Master: aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959)
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:29:58.741955 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260923 19:29:58.767347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37425
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:59.095988 6863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:59.096266 6863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:59.096356 6863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:59.103427 6863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:59.103550 6863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:59.103662 6863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:29:59.113221 6863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:29:59.114523 6863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:59.115748 6863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:59.129026 6869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:59.129067 6868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:59.129088 6871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:59.130817 6863 server_base.cc:1034] running on GCE node
I20260923 19:29:59.131400 6863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:59.132360 6863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:59.133581 6863 hybrid_clock.cc:648] HybridClock initialized: now 1790191799133518 us; error 54 us; skew 500 ppm
I20260923 19:29:59.133898 6863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:59 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191799, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:59.140321 6863 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 19:29:59.142082 6863 webserver.cc:460] Webserver started at http://127.0.97.3:41895/ using document root <none> and password file <none>
I20260923 19:29:59.142860 6863 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:59.142982 6863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:59.143262 6863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:59.146153 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "130dae76d5ba48beb6873ed7e28d8d13"
format_stamp: "Formatted at 2026-09-23 19:29:59 on dist-test-slave-kxfb"
server_key: "cc90d8bc43613603c88266e2c96f78d5"
server_key_iv: "fcf34b8bed4cc906e6a19ab4de85a582"
server_key_version: "encryptionkey@0"
I20260923 19:29:59.146950 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "130dae76d5ba48beb6873ed7e28d8d13"
format_stamp: "Formatted at 2026-09-23 19:29:59 on dist-test-slave-kxfb"
server_key: "cc90d8bc43613603c88266e2c96f78d5"
server_key_iv: "fcf34b8bed4cc906e6a19ab4de85a582"
server_key_version: "encryptionkey@0"
I20260923 19:29:59.152247 6863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 19:29:59.155920 6878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:59.157505 6863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:29:59.157660 6863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "130dae76d5ba48beb6873ed7e28d8d13"
format_stamp: "Formatted at 2026-09-23 19:29:59 on dist-test-slave-kxfb"
server_key: "cc90d8bc43613603c88266e2c96f78d5"
server_key_iv: "fcf34b8bed4cc906e6a19ab4de85a582"
server_key_version: "encryptionkey@0"
I20260923 19:29:59.157872 6863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:59.171157 6863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:59.175848 6863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:59.176239 6863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:59.177266 6863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:29:59.178674 6863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:29:59.178750 6863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:59.178856 6863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:29:59.178906 6863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:59.199137 6723 heartbeater.cc:503] Master 127.0.97.62:37425 was elected leader, sending a full tablet report...
I20260923 19:29:59.215689 6863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37351
I20260923 19:29:59.215797 6990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37351 every 8 connection(s)
I20260923 19:29:59.216987 6863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:29:59.227451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 6863
I20260923 19:29:59.227705 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:29:59.228152 388 external_mini_cluster.cc:1428] Setting key e6baf296694b1c29e2a84cc8e34552ff
Sep 23 19:29:59 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191799, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:29:59.237649 6859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.220146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38897 (local address 127.0.97.62:37425)
0923 19:29:59.220373 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:59.220380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:59.221901 (+ 1521us) server_negotiation.cc:407] Connection header received
0923 19:29:59.223788 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:59.223793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:59.223885 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:59.224003 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:59.225551 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.226465 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.228190 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.228660 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.228780 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.232489 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:59.232493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:59.232497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:59.232527 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:59.234892 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.235537 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.235542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.235545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.235666 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.236042 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.236047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.236050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.236324 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:59.236448 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:59.237113 (+ 665us) server_negotiation.cc:299] Negotiation successful
0923 19:29:59.237333 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 19:29:59.238651 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.220778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.3:38897)
0923 19:29:59.221713 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:59.221761 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:59.223494 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:59.224218 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:59.224227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:59.224763 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:59.225363 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.225379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.226629 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.226634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.228030 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.228042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.228905 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.228913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.229154 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.229836 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:59.229857 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:59.232289 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:59.235015 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.235024 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.235039 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.235427 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.235763 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.235768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.235771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.235941 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.236449 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:59.236457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:59.236770 (+ 313us) client_negotiation.cc:769] Sending connection context
0923 19:29:59.237001 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 19:29:59.237502 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":344,"threads_started":1}
I20260923 19:29:59.240185 6991 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37425
I20260923 19:29:59.240638 6991 heartbeater.cc:464] Registering TS with master...
I20260923 19:29:59.241561 6991 heartbeater.cc:511] Master 127.0.97.62:37425 requested a full tablet report, sending...
I20260923 19:29:59.243397 6534 ts_manager.cc:194] Registered new tserver with Master: 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351)
I20260923 19:29:59.243969 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:29:59.244335 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38897
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260923 19:29:59.294637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35879
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37425,127.0.97.61:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:29:59.659754 6996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:29:59.660131 6996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:29:59.660241 6996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:29:59.668074 6996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:29:59.668205 6996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:29:59.668267 6996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:29:59.668329 6996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:29:59.668376 6996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:29:59.679013 6996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37425,127.0.97.61:35879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
W20260923 19:29:59.679409 6996 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:37425,127.0.97.61:35879'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 19:29:59.680668 6996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:29:59.682147 6996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:29:59.694962 7004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:59.695037 7002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:29:59.695096 7001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:29:59.695912 6996 server_base.cc:1034] running on GCE node
I20260923 19:29:59.696794 6996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:29:59.698408 6996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:29:59.699682 6996 hybrid_clock.cc:648] HybridClock initialized: now 1790191799699590 us; error 68 us; skew 500 ppm
I20260923 19:29:59.700037 6996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:29:59 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191799, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:29:59.703603 6996 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260923 19:29:59.705351 6996 webserver.cc:460] Webserver started at http://127.0.97.61:42341/ using document root <none> and password file <none>
I20260923 19:29:59.706151 6996 fs_manager.cc:359] Metadata directory not provided
I20260923 19:29:59.706274 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:29:59.706648 6996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:29:59.710255 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data/instance:
uuid: "91b529dd056e4b7aa9933aa1c68d339f"
format_stamp: "Formatted at 2026-09-23 19:29:59 on dist-test-slave-kxfb"
server_key: "9cffe006179d368653a3824e53679fd4"
server_key_iv: "f8ee2742d3a62679fbad35eefcb625c1"
server_key_version: "encryptionkey@0"
I20260923 19:29:59.711374 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal/instance:
uuid: "91b529dd056e4b7aa9933aa1c68d339f"
format_stamp: "Formatted at 2026-09-23 19:29:59 on dist-test-slave-kxfb"
server_key: "9cffe006179d368653a3824e53679fd4"
server_key_iv: "f8ee2742d3a62679fbad35eefcb625c1"
server_key_version: "encryptionkey@0"
I20260923 19:29:59.718449 6996 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260923 19:29:59.722572 7011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:29:59.724377 6996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:29:59.724604 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal
uuid: "91b529dd056e4b7aa9933aa1c68d339f"
format_stamp: "Formatted at 2026-09-23 19:29:59 on dist-test-slave-kxfb"
server_key: "9cffe006179d368653a3824e53679fd4"
server_key_iv: "f8ee2742d3a62679fbad35eefcb625c1"
server_key_version: "encryptionkey@0"
I20260923 19:29:59.724869 6996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:29:59.744906 6857 heartbeater.cc:503] Master 127.0.97.62:37425 was elected leader, sending a full tablet report...
I20260923 19:29:59.747314 6996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:29:59.752075 6996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:29:59.752315 6996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:29:59.771409 6996 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35879
I20260923 19:29:59.771472 7062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35879 every 8 connection(s)
I20260923 19:29:59.772917 6996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
I20260923 19:29:59.776479 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 6996
I20260923 19:29:59.777223 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:29:59.787030 7063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:29:59 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 19:29:59 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191799, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:29:59.803169 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.778962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.0.97.61:35879)
0923 19:29:59.779833 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:59.779847 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:59.779898 (+ 51us) server_negotiation.cc:407] Connection header received
0923 19:29:59.781997 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:59.782022 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:59.782553 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:59.782950 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:59.784751 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.786152 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.787951 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.788808 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.788958 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.793970 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:59.793978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:59.793990 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:59.794030 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:59.797654 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.798739 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.798747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.798757 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.798975 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.799446 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.799449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.799451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.800084 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:59.800341 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:59.800947 (+ 606us) server_negotiation.cc:299] Negotiation successful
0923 19:29:59.801467 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":395,"threads_started":1}
I20260923 19:29:59.807289 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.790205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:53774)
0923 19:29:59.791122 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:59.791169 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:59.791323 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:59.792062 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:59.792070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:59.792865 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:59.793240 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.793253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.794432 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.794438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.796172 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.796185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.796991 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.796998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.797184 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.797842 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:59.797860 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:59.800640 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:59.804327 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.804337 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.804344 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.804901 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.805373 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.805376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.805378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.805501 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.806256 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:59.806264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:59.806488 (+ 224us) client_negotiation.cc:769] Sending connection context
0923 19:29:59.806718 (+ 230us) client_negotiation.cc:240] Negotiation successful
0923 19:29:59.806984 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":477,"threads_started":1}
I20260923 19:29:59.807395 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.790238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.0.97.62:37425)
0923 19:29:59.791732 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:59.791736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:59.791752 (+ 16us) server_negotiation.cc:407] Connection header received
0923 19:29:59.791807 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:59.791811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:59.791873 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:59.791971 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:59.793467 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.794247 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.796342 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.796794 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.796899 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.800911 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:59.800919 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:59.800924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:59.800973 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:59.804175 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.805039 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.805046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.805049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.805245 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.805606 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.805611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.805614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.806066 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:59.806237 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:59.806848 (+ 611us) server_negotiation.cc:299] Negotiation successful
0923 19:29:59.807058 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1382,"thread_start_us":1241,"threads_started":1}
I20260923 19:29:59.809077 7063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:29:59 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191799, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 19:29:59.821254 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.809691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.0.1:59226)
0923 19:29:59.809894 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:59.809911 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:59.810023 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:59.810420 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:59.810423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:59.810722 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:59.810864 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.810871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.811816 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.811820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.812720 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.812729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.813486 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.813490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.813617 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.814128 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:59.814144 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:59.816291 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:59.819090 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.819094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.819097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.819516 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.819850 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.819853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.819856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.819983 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.820442 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:59.820445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:59.820569 (+ 124us) client_negotiation.cc:769] Sending connection context
0923 19:29:59.820693 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 19:29:59.820980 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 19:29:59.821534 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.809930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.0.97.61:35879)
0923 19:29:59.810106 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:59.810110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:59.810125 (+ 15us) server_negotiation.cc:407] Connection header received
0923 19:29:59.810183 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:59.810186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:59.810246 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:59.810387 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:59.810982 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.811712 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.812869 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.813344 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.813463 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.816498 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:59.816503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:59.816507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:59.816536 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:59.818970 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.819621 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.819625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.819627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.819741 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.820074 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.820077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.820079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.820346 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:59.820455 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:59.820824 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 19:29:59.821133 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":70}
I20260923 19:29:59.827147 7063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: Bootstrap starting.
I20260923 19:29:59.830926 7063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: Neither blocks nor log segments found. Creating new log.
I20260923 19:29:59.832063 7063 log.cc:824] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: Log is configured to *not* fsync() on all Append() calls
I20260923 19:29:59.835099 7063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: No bootstrap required, opened a new log
I20260923 19:29:59.842514 7063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } }
I20260923 19:29:59.842880 7063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:29:59.842957 7063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b529dd056e4b7aa9933aa1c68d339f, State: Initialized, Role: FOLLOWER
I20260923 19:29:59.843568 7063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } }
I20260923 19:29:59.844530 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } } }
I20260923 19:29:59.844916 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:29:59.845547 7063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: configured and running, proceeding with master startup.
W20260923 19:29:59.854769 7081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:29:59.854900 7081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:29:59.858052 6996 master_runner.cc:428] Detected that this master 91b529dd056e4b7aa9933aa1c68d339f is joining an existing cluster
I20260923 19:29:59.858167 6996 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:35879
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260923 19:29:59.881805 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.865713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.0.97.62:37425)
0923 19:29:59.865917 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:59.865923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:59.867391 (+ 1468us) server_negotiation.cc:407] Connection header received
0923 19:29:59.867667 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:59.867673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:59.867760 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:59.867849 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:59.869313 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.870340 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.872193 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.872751 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.872834 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.875411 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:59.875417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:59.875422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:59.875459 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:59.878730 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.879504 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.879509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.879513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.879657 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.880112 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.880117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.880120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.880475 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:59.880598 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:59.881236 (+ 638us) server_negotiation.cc:299] Negotiation successful
0923 19:29:59.881451 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 19:29:59.882896 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.865057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:53780)
0923 19:29:59.867269 (+ 2212us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:59.867289 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:59.867517 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:59.868085 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:59.868089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:59.868732 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:59.869156 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.869168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.870508 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.870512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.872051 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.872064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.873126 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.873131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.873297 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.874077 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:59.874097 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:59.875206 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:59.878880 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.878884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.878889 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.879369 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.879805 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.879810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.879814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.879986 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.880702 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:59.880707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:59.880946 (+ 239us) client_negotiation.cc:769] Sending connection context
0923 19:29:59.882073 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0923 19:29:59.882436 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2078,"thread_start_us":1622,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260923 19:29:59.898509 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.866562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.0.97.61:35879)
0923 19:29:59.866923 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:59.866929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:59.881390 (+ 14461us) server_negotiation.cc:407] Connection header received
0923 19:29:59.881930 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:59.881936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:59.882016 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:59.882116 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:59.883446 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.884210 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.885661 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.886185 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.886325 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.888563 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:59.888569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:59.888574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:59.888621 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:59.892329 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.893195 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.893201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.893204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.893362 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.895299 (+ 1937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.895305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.895308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.895754 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:59.895917 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:59.896358 (+ 441us) server_negotiation.cc:299] Negotiation successful
0923 19:29:59.896595 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 19:29:59.901306 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.866119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.0.1:59238)
0923 19:29:59.881231 (+ 15112us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:59.881260 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:59.881770 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:59.882258 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:59.882263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:59.882990 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:59.883301 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.883312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.884435 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.884440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.885540 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.885550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.886638 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.886644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.886832 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.887369 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:59.887391 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:59.888330 (+ 939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:59.892498 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.892503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.892507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.893062 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.893496 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.893503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.893506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.893654 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.895985 (+ 2331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:59.895989 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:59.896122 (+ 133us) client_negotiation.cc:769] Sending connection context
0923 19:29:59.896865 (+ 743us) client_negotiation.cc:240] Negotiation successful
0923 19:29:59.898758 (+ 1893us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14912,"thread_start_us":15279,"threads_started":1}
Sep 23 19:29:59 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191797, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 19:29:59.916441 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.890199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.0.1:59254)
0923 19:29:59.896385 (+ 6186us) negotiation.cc:107] Waiting for socket to connect
0923 19:29:59.896443 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 19:29:59.896660 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:29:59.899239 (+ 2579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:29:59.899249 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:29:59.900447 (+ 1198us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:29:59.900765 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.900778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.902355 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.902362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.903995 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:29:59.904010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.905099 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.905105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:29:59.905315 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.906079 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:29:59.906104 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:29:59.909246 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:29:59.913523 (+ 4277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.913537 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.913543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.914037 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.914524 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:29:59.914529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:29:59.914533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:29:59.914756 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:29:59.915499 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:29:59.915505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:29:59.915684 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 19:29:59.915831 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 19:29:59.916127 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5840,"thread_start_us":5645,"threads_started":1}
I20260923 19:29:59.916818 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:29:59.895469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.0.97.61:35879)
0923 19:29:59.898776 (+ 3307us) server_negotiation.cc:206] Beginning negotiation
0923 19:29:59.898782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:29:59.898803 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:29:59.898907 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:29:59.898913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:29:59.899012 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:29:59.899131 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:29:59.900948 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.902149 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.904165 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:29:59.904901 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:29:59.905024 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:29:59.909473 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:29:59.909481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:29:59.909486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:29:59.909528 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:29:59.913297 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.914190 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.914196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.914200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.914381 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:29:59.914900 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:29:59.914905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:29:59.914909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:29:59.915340 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:29:59.915482 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 19:29:59.916112 (+ 630us) server_negotiation.cc:299] Negotiation successful
0923 19:29:59.916375 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3206,"thread_start_us":4132,"threads_started":1}
I20260923 19:29:59.918856 6534 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:35879: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 35879
}
attrs {
promote: true
}
}
dest_uuid: "164e149b0070471e81a429e1c5fe1947"
cas_config_opid_index: -1
I20260923 19:29:59.919844 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260923 19:29:59.922457 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44187
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37425,127.0.97.61:35879,127.0.97.60:44187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:29:59.935707 7038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:29:59.935155 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 91b529dd056e4b7aa9933aa1c68d339f (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } } }
I20260923 19:29:59.938690 7038 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Refusing update from remote peer 164e149b0070471e81a429e1c5fe1947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 19:29:59.942487 7094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:29:59.954995 7095 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:35879
I20260923 19:29:59.955947 6996 master.cc:537] Master@127.0.97.61:35879 shutting down...
I20260923 19:29:59.956171 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } } }
I20260923 19:29:59.956506 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:29:59.976382 7038 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 91b529dd056e4b7aa9933aa1c68d339f (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } } }
I20260923 19:29:59.982211 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164e149b0070471e81a429e1c5fe1947. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } } }
I20260923 19:29:59.982256 7101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } } }
I20260923 19:29:59.982548 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: LEARNER
I20260923 19:29:59.982584 7101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: LEARNER
I20260923 19:29:59.982805 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } } }
I20260923 19:29:59.983057 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: LEARNER
I20260923 19:30:00.001086 6996 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 LEARNER]: Raft consensus shutting down.
I20260923 19:30:00.001782 6996 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 LEARNER]: Raft consensus is shut down!
I20260923 19:30:00.001969 6996 tablet_replica.cc:330] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: stopping tablet replica
I20260923 19:30:00.021437 6996 master.cc:552] Master@127.0.97.61:35879 shutdown complete.
I20260923 19:30:00.021625 6996 master_runner.cc:305] Clearing existing system tablet
I20260923 19:30:00.022118 6996 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:00.022365 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:00.026459 6996 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 19:30:00.028602 7106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:00.029064 6996 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:00.029557 6996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260923 19:30:00.029709 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal
uuid: "91b529dd056e4b7aa9933aa1c68d339f"
format_stamp: "Formatted at 2026-09-23 19:29:59 on dist-test-slave-kxfb"
server_key: "9cffe006179d368653a3824e53679fd4"
server_key_iv: "f8ee2742d3a62679fbad35eefcb625c1"
server_key_version: "encryptionkey@0"
I20260923 19:30:00.034767 6996 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:00.042048 6996 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 19:30:00.042420 6996 log.cc:1196] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 19:30:00.043036 6996 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: Deleting consensus metadata
I20260923 19:30:00.043417 6996 master_runner.cc:315] Copying system tablet from 127.0.97.62:37425
I20260923 19:30:00.046487 6996 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:37425
I20260923 19:30:00.063622 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.047577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.0.97.62:37425)
0923 19:30:00.047780 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.047787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.048448 (+ 661us) server_negotiation.cc:407] Connection header received
0923 19:30:00.048721 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.048727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.048821 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.048921 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.050143 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.051281 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.053281 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.053915 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.054047 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.056610 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.056616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.056622 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.056662 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.060278 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.061182 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.061188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.061192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.061362 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.061802 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.061807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.061810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.062187 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.062438 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.062966 (+ 528us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.063204 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 19:30:00.063830 7112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.047547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:53796)
0923 19:30:00.048251 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.048304 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.048572 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.049137 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.049141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.049694 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:00.049963 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.049975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.051471 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.051477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.053071 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.053089 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.054118 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.054124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.054362 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.055165 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:00.055190 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:00.056394 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:00.060437 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.060443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.060448 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.061019 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.061515 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.061520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.061524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.061682 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.062400 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:00.062405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:00.062636 (+ 231us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.062836 (+ 200us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.063298 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":376,"threads_started":1}
I20260923 19:30:00.064993 6554 tablet_copy_service.cc:140] P 164e149b0070471e81a429e1c5fe1947: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 91b529dd056e4b7aa9933aa1c68d339f ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:53796)
I20260923 19:30:00.065326 6554 tablet_copy_service.cc:161] P 164e149b0070471e81a429e1c5fe1947: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 91b529dd056e4b7aa9933aa1c68d339f at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:53796: session id = 91b529dd056e4b7aa9933aa1c68d339f-00000000000000000000000000000000
I20260923 19:30:00.070340 6554 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 19:30:00.073385 6996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 19:30:00.080541 6996 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: tablet copy: Starting download of 0 data blocks...
I20260923 19:30:00.080856 6996 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: tablet copy: Starting download of 1 WAL segments...
I20260923 19:30:00.084128 6996 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 19:30:00.088724 6554 tablet_copy_service.cc:342] P 164e149b0070471e81a429e1c5fe1947: Request end of tablet copy session 91b529dd056e4b7aa9933aa1c68d339f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:53796
I20260923 19:30:00.089021 6554 tablet_copy_service.cc:433] P 164e149b0070471e81a429e1c5fe1947: ending tablet copy session 91b529dd056e4b7aa9933aa1c68d339f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 91b529dd056e4b7aa9933aa1c68d339f
I20260923 19:30:00.101531 6996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:00.104784 7113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:00.104983 7114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:00.112133 7116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:00.112413 6996 server_base.cc:1034] running on GCE node
I20260923 19:30:00.112870 6996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:00.113596 6996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:00.114930 6996 hybrid_clock.cc:648] HybridClock initialized: now 1790191800114875 us; error 29 us; skew 500 ppm
I20260923 19:30:00.115161 6996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:00 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191800, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:00.117158 6996 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260923 19:30:00.118566 6996 webserver.cc:460] Webserver started at http://127.0.97.61:34603/ using document root <none> and password file <none>
I20260923 19:30:00.118902 6996 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:00.118974 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:00.121618 6996 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260923 19:30:00.123524 7123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:00.123888 6996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 19:30:00.124006 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal
uuid: "91b529dd056e4b7aa9933aa1c68d339f"
format_stamp: "Formatted at 2026-09-23 19:29:59 on dist-test-slave-kxfb"
server_key: "9cffe006179d368653a3824e53679fd4"
server_key_iv: "f8ee2742d3a62679fbad35eefcb625c1"
server_key_version: "encryptionkey@0"
I20260923 19:30:00.124159 6996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:00.139709 6996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:00.143483 6996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:00.195444 6996 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35879
I20260923 19:30:00.195462 7174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35879 every 8 connection(s)
I20260923 19:30:00.196576 6996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
I20260923 19:30:00.203442 7175 sys_catalog.cc:263] Verifying existing consensus state
I20260923 19:30:00.208884 7175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: Bootstrap starting.
I20260923 19:30:00.231925 7175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 19:30:00.232465 7175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: Bootstrap complete.
I20260923 19:30:00.233383 7175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } }
I20260923 19:30:00.233629 7175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91b529dd056e4b7aa9933aa1c68d339f, State: Initialized, Role: LEARNER
I20260923 19:30:00.233886 7175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } }
I20260923 19:30:00.235083 7175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:30:00.234838 7178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } } }
I20260923 19:30:00.235357 7178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: LEARNER
I20260923 19:30:00.246673 7189 catalog_manager.cc:1259] Loaded cluster ID: 7cf1765f85b34a0d83e9aacd359ae234
I20260923 19:30:00.246819 7189 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: loading cluster ID for follower catalog manager: success
I20260923 19:30:00.247479 6991 heartbeater.cc:503] Master 127.0.97.62:37425 was elected leader, sending a full tablet report...
I20260923 19:30:00.251893 7189 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: acquiring CA information for follower catalog manager: success
I20260923 19:30:00.254810 7189 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 19:30:00.365899 7096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:00.366200 7096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:00.366361 7096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:00.373744 7096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:30:00.373854 7096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:00.373929 7096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:00.373998 7096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:30:00.374064 7096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:30:00.383819 7096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37425,127.0.97.61:35879,127.0.97.60:44187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:00.385195 7096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:00.386462 7096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:00.399290 7195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:00.399391 7197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:00.399259 7194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:00.400066 7096 server_base.cc:1034] running on GCE node
I20260923 19:30:00.400985 7096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:00.402770 7096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:00.403999 7096 hybrid_clock.cc:648] HybridClock initialized: now 1790191800403948 us; error 55 us; skew 500 ppm
I20260923 19:30:00.404386 7096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:00 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191800, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:00.408334 7096 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260923 19:30:00.410182 7096 webserver.cc:460] Webserver started at http://127.0.97.60:43073/ using document root <none> and password file <none>
I20260923 19:30:00.411036 7096 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:00.411898 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:00.412254 7096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:00.415726 7096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data/instance:
uuid: "0d25d7bc72b14cac94ddb530ede55d42"
format_stamp: "Formatted at 2026-09-23 19:30:00 on dist-test-slave-kxfb"
server_key: "bdc69b0b52e8eeff2f7cc615e53e986c"
server_key_iv: "78f70d506363392d4e6b050f02a1f9e7"
server_key_version: "encryptionkey@0"
I20260923 19:30:00.416594 7096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal/instance:
uuid: "0d25d7bc72b14cac94ddb530ede55d42"
format_stamp: "Formatted at 2026-09-23 19:30:00 on dist-test-slave-kxfb"
server_key: "bdc69b0b52e8eeff2f7cc615e53e986c"
server_key_iv: "78f70d506363392d4e6b050f02a1f9e7"
server_key_version: "encryptionkey@0"
I20260923 19:30:00.423034 7096 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 19:30:00.427685 7204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:00.429519 7096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:30:00.429751 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal
uuid: "0d25d7bc72b14cac94ddb530ede55d42"
format_stamp: "Formatted at 2026-09-23 19:30:00 on dist-test-slave-kxfb"
server_key: "bdc69b0b52e8eeff2f7cc615e53e986c"
server_key_iv: "78f70d506363392d4e6b050f02a1f9e7"
server_key_version: "encryptionkey@0"
I20260923 19:30:00.429975 7096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:00.458526 7096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:00.463413 7096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:00.463701 7096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:00.484683 7096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44187
I20260923 19:30:00.484688 7255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44187 every 8 connection(s)
I20260923 19:30:00.486634 7096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
I20260923 19:30:00.491807 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:00.496861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 7096
I20260923 19:30:00.498340 7256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:30:00 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191800, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 23 19:30:00 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 19:30:00.520375 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.501432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.0.97.62:37425)
0923 19:30:00.501646 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.501653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.502341 (+ 688us) server_negotiation.cc:407] Connection header received
0923 19:30:00.502818 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.502824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.502906 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.503000 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.505190 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.506339 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.508534 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.509118 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.509255 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.513339 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.513345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.513349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.513388 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.516311 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.517503 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.517508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.517511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.517672 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.518280 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.518315 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.518320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.518677 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.518828 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.519691 (+ 863us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.519938 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 19:30:00.521643 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.499146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.0.97.60:44187)
0923 19:30:00.500048 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.500066 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.500120 (+ 54us) server_negotiation.cc:407] Connection header received
0923 19:30:00.502371 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.502405 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.502741 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.503008 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.504888 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.506699 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.508432 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.509275 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.509452 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.514129 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.514139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.514151 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.514180 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.517443 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.518014 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.518021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.518024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.518152 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.518585 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.518589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.518591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.519124 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.519445 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.519751 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.520160 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":637,"thread_start_us":348,"threads_started":1}
I20260923 19:30:00.521687 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.501234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:53810)
0923 19:30:00.502165 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.502217 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.502589 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.503085 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.503091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.504506 (+ 1415us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:00.505015 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.505038 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.506511 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.506516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.508398 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.508413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.509335 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.509342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.509565 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.510362 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:00.510393 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:00.513090 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:00.516459 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.516469 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.516482 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.517363 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.517822 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.517829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.517834 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.518129 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.518820 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:00.518831 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:00.519192 (+ 361us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.519588 (+ 396us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.520343 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":125,"spinlock_wait_cycles":22400,"thread_start_us":453,"threads_started":1}
I20260923 19:30:00.523459 7256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:30:00 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191800, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 19:30:00.533730 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } } }
I20260923 19:30:00.534135 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:00.540241 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.524441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.0.1:59266)
0923 19:30:00.524762 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.524779 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.524869 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.525218 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.525221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.525579 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:00.525758 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.525767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.527180 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.527184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.528313 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.528325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.528987 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.528991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.529117 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.529716 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:00.529733 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:00.531793 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:00.537048 (+ 5255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.537056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.537061 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.537676 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.538181 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.538187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.538191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.538403 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.539088 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:00.539095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:00.539292 (+ 197us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.539425 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.539812 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260923 19:30:00.541699 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.524370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.0.97.61:35879)
0923 19:30:00.524960 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.524964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.524980 (+ 16us) server_negotiation.cc:407] Connection header received
0923 19:30:00.525028 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.525032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.525089 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.525180 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.525900 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.527025 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.528445 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.528863 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.528971 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.532013 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.532018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.532022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.532052 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.536826 (+ 4774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.537869 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.537876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.537880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.538052 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.538550 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.538555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.538558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.538940 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.540818 (+ 1878us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.540960 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.541242 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":317,"threads_started":1}
I20260923 19:30:00.543210 7256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:30:00.548513 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.534214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.0.97.61:35879)
0923 19:30:00.536062 (+ 1848us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.536069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.542093 (+ 6024us) server_negotiation.cc:407] Connection header received
0923 19:30:00.542360 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.542364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.542468 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.542620 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:00.543110 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.544130 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.546205 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.547568 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.547721 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.548057 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.548103 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.548214 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":315,"threads_started":1}
I20260923 19:30:00.548616 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.533860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.0.1:59268)
0923 19:30:00.541941 (+ 8081us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.541963 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.542154 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.542668 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.542671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.542693 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:00.542939 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.542951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.544276 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.544280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.546058 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.546072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.547755 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.547759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.547917 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.547980 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.548061 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.548211 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7951,"thread_start_us":7823,"threads_started":1}
I20260923 19:30:00.549698 7178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164e149b0070471e81a429e1c5fe1947. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: true } } }
I20260923 19:30:00.550012 7178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: LEARNER
I20260923 19:30:00.550634 7264 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: attempting to promote NON_VOTER 91b529dd056e4b7aa9933aa1c68d339f to VOTER
Sep 23 19:30:00 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191800, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 19:30:00.551739 7264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } }
I20260923 19:30:00.554565 7150 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 LEARNER]: Refusing update from remote peer 164e149b0070471e81a429e1c5fe1947: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 19:30:00.555214 7263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 19:30:00.558974 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.543983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44187 (local address 127.0.0.1:46382)
0923 19:30:00.544270 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.544289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.544378 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.544696 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.544699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.545063 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:00.545228 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.545236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.546476 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.546482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.547778 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.547791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.548988 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.548994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.549206 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.549890 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:00.549914 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:00.552585 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:00.556100 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.556106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.556111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.556602 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.557065 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.557069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.557071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.557248 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.557857 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:00.557863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:00.558041 (+ 178us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.558194 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.558543 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260923 19:30:00.559011 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.544032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.0.97.60:44187)
0923 19:30:00.544180 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.544184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.544360 (+ 176us) server_negotiation.cc:407] Connection header received
0923 19:30:00.544484 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.544488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.544565 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.544673 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.545389 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.546360 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.547926 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.548503 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.548882 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.552760 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.552764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.552768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.552796 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.555930 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.556758 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.556764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.556768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.556930 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.557369 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.557375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.557379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.557713 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.557824 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.558322 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.558681 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":52}
I20260923 19:30:00.558966 7262 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91b529dd056e4b7aa9933aa1c68d339f (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } }
I20260923 19:30:00.562463 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } }
I20260923 19:30:00.562738 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:00.563148 7150 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91b529dd056e4b7aa9933aa1c68d339f (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } }
I20260923 19:30:00.564904 7256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Bootstrap starting.
I20260923 19:30:00.566658 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } }
I20260923 19:30:00.566952 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:00.566803 7178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } }
I20260923 19:30:00.567061 7178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:30:00.568922 7256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:00.569839 7256 log.cc:824] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:00.572641 7256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: No bootstrap required, opened a new log
I20260923 19:30:00.579782 7256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } }
I20260923 19:30:00.580111 7256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:00.580183 7256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d25d7bc72b14cac94ddb530ede55d42, State: Initialized, Role: FOLLOWER
I20260923 19:30:00.580681 7256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } }
I20260923 19:30:00.581717 7270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } } }
I20260923 19:30:00.582360 7270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:30:00.582759 7256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 19:30:00.593943 7281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:30:00.594122 7281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:30:00.597112 7096 master_runner.cc:428] Detected that this master 0d25d7bc72b14cac94ddb530ede55d42 is joining an existing cluster
I20260923 19:30:00.597280 7096 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:44187
I20260923 19:30:00.597136 7270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:00.597530 7270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } }
I20260923 19:30:00.599259 7270 leader_election.cc:288] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425), 91b529dd056e4b7aa9933aa1c68d339f (127.0.97.61:35879)
I20260923 19:30:00.600136 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d25d7bc72b14cac94ddb530ede55d42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b529dd056e4b7aa9933aa1c68d339f" is_pre_election: true
I20260923 19:30:00.600474 6544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d25d7bc72b14cac94ddb530ede55d42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164e149b0070471e81a429e1c5fe1947" is_pre_election: true
I20260923 19:30:00.601625 7207 leader_election.cc:302] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d25d7bc72b14cac94ddb530ede55d42; no voters: 164e149b0070471e81a429e1c5fe1947, 91b529dd056e4b7aa9933aa1c68d339f
I20260923 19:30:00.602167 7270 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 19:30:00.628166 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 6595
I20260923 19:30:00.629267 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.612890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.0.1:59278)
0923 19:30:00.613620 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.613642 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.613787 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.614111 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.614116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.614923 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:00.615242 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.615250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.617253 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.617258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.618921 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.618937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.619984 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.619991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.620165 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.620811 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:00.620832 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:00.621936 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:00.625330 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.625335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.625339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.625881 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.626896 (+ 1015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.626902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.626905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.627223 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.627909 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:00.627914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:00.628140 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.628418 (+ 278us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.628765 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":675,"threads_started":1}
I20260923 19:30:00.630362 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.613489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.0.97.61:35879)
0923 19:30:00.613701 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.613707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.613727 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:30:00.613911 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.613915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.613984 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.614062 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.615370 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.616947 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.619075 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.619639 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.619749 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.622146 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.622152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.622158 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.622193 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.625206 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.625987 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.625992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.625996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.626142 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.627369 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.627375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.627379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.627769 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.629621 (+ 1852us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.629748 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.629993 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 19:30:00.631780 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.612730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.0.97.62:37425)
0923 19:30:00.613002 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.613010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.615995 (+ 2985us) server_negotiation.cc:407] Connection header received
0923 19:30:00.616246 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.616252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.616334 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.616402 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.617751 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.618834 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.620634 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.621290 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.621377 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.625394 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.625400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.625405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.625443 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.628844 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.629591 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.629596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.629599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.629737 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.630167 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.630172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.630175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.630526 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.630658 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.631244 (+ 586us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.631466 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 19:30:00.632799 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.614012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:53812)
0923 19:30:00.615865 (+ 1853us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.615888 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.616115 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.616603 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.616607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.617236 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:00.617595 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.617605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.619009 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.619014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.620473 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.620495 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.621688 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.621695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.621906 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.622473 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:00.622507 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:00.625191 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:00.628995 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.628999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.629002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.629480 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.629869 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.629873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.629876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.630041 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.630750 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:00.630754 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:00.630962 (+ 208us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.632006 (+ 1044us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.632339 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1662,"thread_start_us":1474,"threads_started":1}
I20260923 19:30:00.646512 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.616027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.0.97.60:44187)
0923 19:30:00.617177 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.617182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.624840 (+ 7658us) server_negotiation.cc:407] Connection header received
0923 19:30:00.625089 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.625098 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.625166 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.625244 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.632324 (+ 7080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.633387 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.635507 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.636237 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.636342 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.639577 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.639587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.639591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.639623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.643095 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.643826 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.643831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.643835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.643955 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.644300 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.644304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.644308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.644713 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.644836 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.645302 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.645528 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045}
I20260923 19:30:00.647997 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.616173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44187 (local address 127.0.0.1:46396)
0923 19:30:00.624715 (+ 8542us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.624738 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.624949 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.625510 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.625515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.626678 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:00.632170 (+ 5492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.632181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.633620 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.633625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.635316 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.635329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.636675 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.636681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.636847 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.637436 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:00.637457 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:00.639362 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:00.643231 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.643236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.643240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.643718 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.644072 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.644077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.644080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.644203 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.644892 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:00.644896 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:00.645044 (+ 148us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.647241 (+ 2197us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.647522 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7581,"thread_start_us":6684,"threads_started":1}
Sep 23 19:30:00 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
Sep 23 19:30:00 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191797, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 19:30:00.659689 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.629446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.0.97.1:43027)
0923 19:30:00.638576 (+ 9130us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.638586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.638619 (+ 33us) server_negotiation.cc:407] Connection header received
0923 19:30:00.638691 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.638700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.639023 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.640775 (+ 1752us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.641020 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.642563 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.644335 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.645200 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.645841 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.652354 (+ 6513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.652367 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.652387 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.652428 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.656294 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.657123 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.657132 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.657135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.657275 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.657700 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.657705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.657709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.658129 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.658465 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.658966 (+ 501us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.659285 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8948,"thread_start_us":7692,"threads_started":1}
I20260923 19:30:00.661347 6718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:00.666858 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.638179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44187 (local address 127.0.0.1:46406)
0923 19:30:00.638733 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.638753 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.638882 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.647185 (+ 8303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.647192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.647818 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:00.648102 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.648116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.649776 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.649782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.651359 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.651376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.652601 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.652607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.652832 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.653746 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:00.653771 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:00.657668 (+ 3897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:00.663492 (+ 5824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.663499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.663504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.664108 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.664606 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.664612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.664616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.664865 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.665667 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:00.665672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:00.665890 (+ 218us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.666042 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.666415 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400}
I20260923 19:30:00.668027 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.639216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.0.97.60:44187)
0923 19:30:00.646750 (+ 7534us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.646756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.646773 (+ 17us) server_negotiation.cc:407] Connection header received
0923 19:30:00.646861 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.646866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.646974 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.647099 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.648311 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.649571 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.651618 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.652410 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.653253 (+ 843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.657871 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.657876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.657879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.657911 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.663239 (+ 5328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.664283 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.664293 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.664298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.664470 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.664991 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.665000 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.665004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.665424 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.665572 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.667323 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.667589 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7436,"thread_start_us":17221,"threads_started":1}
I20260923 19:30:00.680486 6534 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:44187: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 44187
}
attrs {
promote: true
}
}
dest_uuid: "164e149b0070471e81a429e1c5fe1947"
cas_config_opid_index: 6
I20260923 19:30:00.681450 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } }
I20260923 19:30:00.684784 7150 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Refusing update from remote peer 164e149b0070471e81a429e1c5fe1947: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 19:30:00.685395 7231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:00.685613 7262 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 19:30:00.688337 7263 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0d25d7bc72b14cac94ddb530ede55d42 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } } }
I20260923 19:30:00.688989 7150 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0d25d7bc72b14cac94ddb530ede55d42 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } } }
I20260923 19:30:00.693580 7178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } } }
I20260923 19:30:00.693866 7178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:30:00.694391 7231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Refusing update from remote peer 164e149b0070471e81a429e1c5fe1947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 19:30:00.694798 7263 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:44187
I20260923 19:30:00.695641 7096 master.cc:537] Master@127.0.97.60:44187 shutting down...
I20260923 19:30:00.696086 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } } }
I20260923 19:30:00.696364 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:00.696395 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } } }
I20260923 19:30:00.696729 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:00.701155 7262 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 19:30:00.704895 6520 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 19:30:00.707747 6520 consensus_peers.cc:487] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 -> Peer 0d25d7bc72b14cac94ddb530ede55d42 (127.0.97.60:44187): Couldn't send request to peer 0d25d7bc72b14cac94ddb530ede55d42. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 19:30:00.711661 7096 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:00.712296 7096 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:00.712589 7096 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: stopping tablet replica
I20260923 19:30:00.732002 7096 master.cc:552] Master@127.0.97.60:44187 shutdown complete.
I20260923 19:30:00.732116 7096 master_runner.cc:305] Clearing existing system tablet
I20260923 19:30:00.732513 7096 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:00.732613 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:00.735702 7096 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260923 19:30:00.737454 7295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:00.738041 7096 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:00.738540 7096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 19:30:00.738698 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal
uuid: "0d25d7bc72b14cac94ddb530ede55d42"
format_stamp: "Formatted at 2026-09-23 19:30:00 on dist-test-slave-kxfb"
server_key: "bdc69b0b52e8eeff2f7cc615e53e986c"
server_key_iv: "78f70d506363392d4e6b050f02a1f9e7"
server_key_version: "encryptionkey@0"
I20260923 19:30:00.744119 7096 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:00.751967 7096 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 19:30:00.752257 7096 log.cc:1196] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 19:30:00.752759 7096 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Deleting consensus metadata
I20260923 19:30:00.753049 7096 master_runner.cc:315] Copying system tablet from 127.0.97.62:37425
I20260923 19:30:00.755889 7096 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:37425
I20260923 19:30:00.778661 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.756854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.0.97.62:37425)
0923 19:30:00.757151 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:00.757159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:00.761406 (+ 4247us) server_negotiation.cc:407] Connection header received
0923 19:30:00.761629 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:00.761635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:00.761742 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:00.761889 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:00.763129 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.764240 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.766173 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.767149 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.767346 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.771648 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:00.771656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:00.771661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:00.771702 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:00.775263 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.776157 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.776164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.776167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.776331 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:00.776857 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:00.776862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:00.776866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:00.777259 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:00.777398 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:00.778004 (+ 606us) server_negotiation.cc:299] Negotiation successful
0923 19:30:00.778274 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260923 19:30:00.780094 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:00.756755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:53826)
0923 19:30:00.761242 (+ 4487us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:00.761273 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:00.761479 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:00.762002 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:00.762008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:00.762644 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:00.762953 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.762966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.764413 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.764420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.766004 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:00.766018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:00.767720 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:00.767730 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:00.768063 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:00.769557 (+ 1494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:00.769582 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:00.771347 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:00.775489 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.775494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.775498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.775996 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.776504 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:00.776509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:00.776512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:00.776715 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:00.777484 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:00.777489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:00.777692 (+ 203us) client_negotiation.cc:769] Sending connection context
0923 19:30:00.779238 (+ 1546us) client_negotiation.cc:240] Negotiation successful
0923 19:30:00.779591 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4318,"thread_start_us":4197,"threads_started":1}
I20260923 19:30:00.781008 6554 tablet_copy_service.cc:140] P 164e149b0070471e81a429e1c5fe1947: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0d25d7bc72b14cac94ddb530ede55d42 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53826)
I20260923 19:30:00.781201 6554 tablet_copy_service.cc:161] P 164e149b0070471e81a429e1c5fe1947: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0d25d7bc72b14cac94ddb530ede55d42 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53826: session id = 0d25d7bc72b14cac94ddb530ede55d42-00000000000000000000000000000000
I20260923 19:30:00.784235 6554 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 19:30:00.785833 7096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 19:30:00.792335 7096 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: tablet copy: Starting download of 0 data blocks...
I20260923 19:30:00.792637 7096 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: tablet copy: Starting download of 1 WAL segments...
I20260923 19:30:00.799768 7096 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 19:30:00.804471 6554 tablet_copy_service.cc:342] P 164e149b0070471e81a429e1c5fe1947: Request end of tablet copy session 0d25d7bc72b14cac94ddb530ede55d42-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53826
I20260923 19:30:00.804715 6554 tablet_copy_service.cc:433] P 164e149b0070471e81a429e1c5fe1947: ending tablet copy session 0d25d7bc72b14cac94ddb530ede55d42-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0d25d7bc72b14cac94ddb530ede55d42
I20260923 19:30:00.820188 7096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:00.828662 7302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:00.829106 7303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:00.852566 7096 server_base.cc:1034] running on GCE node
W20260923 19:30:00.853825 7305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:00.854957 7096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:00.855948 7096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:00.856678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 6595
I20260923 19:30:00.857931 7096 hybrid_clock.cc:648] HybridClock initialized: now 1790191800857884 us; error 48 us; skew 500 ppm
I20260923 19:30:00.858249 7096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:00 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191800, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:00.860663 7096 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260923 19:30:00.867748 7096 webserver.cc:460] Webserver started at http://127.0.97.60:36145/ using document root <none> and password file <none>
I20260923 19:30:00.868077 7096 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:00.868194 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:00.871104 7096 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 19:30:00.872984 7312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:00.873392 7096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260923 19:30:00.873519 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal
uuid: "0d25d7bc72b14cac94ddb530ede55d42"
format_stamp: "Formatted at 2026-09-23 19:30:00 on dist-test-slave-kxfb"
server_key: "bdc69b0b52e8eeff2f7cc615e53e986c"
server_key_iv: "78f70d506363392d4e6b050f02a1f9e7"
server_key_version: "encryptionkey@0"
I20260923 19:30:00.873740 7096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:00.880470 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43027
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37425
--webserver_port=34839
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37425,127.0.97.61:35879
--tserver_master_addrs=127.0.97.62:37425,127.0.97.61:35879,127.0.97.60:44187 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:30:00.908581 7096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:00.912150 7096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:00.957000 7096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44187
I20260923 19:30:00.957026 7364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44187 every 8 connection(s)
I20260923 19:30:00.958050 7096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
I20260923 19:30:00.965358 7365 sys_catalog.cc:263] Verifying existing consensus state
I20260923 19:30:00.969910 7365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Bootstrap starting.
I20260923 19:30:00.986074 7365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 19:30:00.986618 7365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Bootstrap complete.
I20260923 19:30:00.987660 7365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } }
I20260923 19:30:00.987924 7365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d25d7bc72b14cac94ddb530ede55d42, State: Initialized, Role: LEARNER
I20260923 19:30:00.988189 7365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } }
I20260923 19:30:00.989090 7368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } } }
I20260923 19:30:00.989424 7365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:30:00.989625 7368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: This master's current role is: LEARNER
I20260923 19:30:01.003837 7379 catalog_manager.cc:1259] Loaded cluster ID: 7cf1765f85b34a0d83e9aacd359ae234
I20260923 19:30:01.003953 7379 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: loading cluster ID for follower catalog manager: success
I20260923 19:30:01.006867 7379 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: acquiring CA information for follower catalog manager: success
I20260923 19:30:01.008991 7379 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 19:30:01.178579 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:01.172647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.0.97.60:44187)
0923 19:30:01.173295 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:01.173300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:01.173316 (+ 16us) server_negotiation.cc:407] Connection header received
0923 19:30:01.173382 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:01.173386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:01.173460 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:01.173571 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:01.174015 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.175140 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.176733 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.177541 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.177652 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:01.177932 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:01.178047 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 19:30:01.178181 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":337,"threads_started":1}
I20260923 19:30:01.178579 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:01.172369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44187 (local address 127.0.0.1:46412)
0923 19:30:01.172904 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:01.172929 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:01.173105 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:01.173619 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:01.173623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:01.173649 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:01.173877 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:01.173887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.175295 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.175300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:01.176588 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:01.176600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.177685 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.177690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:01.177888 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:01.177926 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 19:30:01.178023 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 19:30:01.178171 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":282,"threads_started":1}
I20260923 19:30:01.179862 7340 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260923 19:30:01.180536 7368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164e149b0070471e81a429e1c5fe1947. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } } }
I20260923 19:30:01.180907 7368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: This master's current role is: LEARNER
W20260923 19:30:01.290181 7317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:01.290583 7317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:01.290714 7317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:01.298552 7317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:01.298692 7317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:01.298839 7317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:30:01.308723 7317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34839
--tserver_master_addrs=127.0.97.62:37425,127.0.97.61:35879,127.0.97.60:44187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:01.310086 7317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:01.311391 7317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:01.325896 7386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:01.326174 7389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:01.325970 7387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:01.327031 7317 server_base.cc:1034] running on GCE node
I20260923 19:30:01.327675 7317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:01.328841 7317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:01.330054 7317 hybrid_clock.cc:648] HybridClock initialized: now 1790191801329982 us; error 69 us; skew 500 ppm
I20260923 19:30:01.330430 7317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:01 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191801, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:01.333863 7317 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 19:30:01.335541 7317 webserver.cc:460] Webserver started at http://127.0.97.1:34839/ using document root <none> and password file <none>
I20260923 19:30:01.336227 7317 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:01.336369 7317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:01.342869 7317 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:30:01.346479 7396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:01.348207 7317 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:30:01.348410 7317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "066538205b95484ca562199aa7740b8a"
format_stamp: "Formatted at 2026-09-23 19:29:58 on dist-test-slave-kxfb"
server_key: "5abeaf0b462df250c7d9c0d54bf5cc8b"
server_key_iv: "1d6ed920b662b3e8acc97c57e37e1957"
server_key_version: "encryptionkey@0"
I20260923 19:30:01.349579 7317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:01.371955 7317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:01.377028 7317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:01.377491 7317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:01.378794 7317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:01.380564 7317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:01.380693 7317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:01.380824 7317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:01.380901 7317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:01.420779 7317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43027
I20260923 19:30:01.420848 7508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43027 every 8 connection(s)
I20260923 19:30:01.422561 7317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:30:01.432145 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 7317
I20260923 19:30:01.432439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 6729
Sep 23 19:30:01 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191801, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 23 19:30:01 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191801, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 23 19:30:01 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191801, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 19:30:01.454854 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:01.426052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60247 (local address 127.0.97.60:44187)
0923 19:30:01.426259 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:01.426266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:01.428190 (+ 1924us) server_negotiation.cc:407] Connection header received
0923 19:30:01.431309 (+ 3119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:01.431314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:01.431392 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:01.431717 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:01.434658 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.435834 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.438961 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.439690 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.439787 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:01.446149 (+ 6362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:01.446158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:01.446162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:01.446210 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:01.450527 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:01.451692 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:01.451699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:01.451704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:01.451858 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:01.452411 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:01.452416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:01.452419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:01.452794 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:01.452935 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:01.454107 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0923 19:30:01.454489 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 19:30:01.456467 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:01.428876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:33581)
0923 19:30:01.429552 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:01.429569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:01.430867 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:01.431698 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:01.431712 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:01.432798 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:01.434394 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:01.434426 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.436913 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.436919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:01.438781 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:01.438797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.439946 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.439957 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:01.440317 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:01.441367 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:01.441416 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:01.445192 (+ 3776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:01.450932 (+ 5740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:01.450937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:01.450941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:01.451817 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:01.452241 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:01.452246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:01.452249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:01.452407 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:01.453020 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:01.453030 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:01.453634 (+ 604us) client_negotiation.cc:769] Sending connection context
0923 19:30:01.453985 (+ 351us) client_negotiation.cc:240] Negotiation successful
0923 19:30:01.454683 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":296,"spinlock_wait_cycles":645120,"thread_start_us":343,"threads_started":1}
I20260923 19:30:01.456485 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:01.426982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44187 (local address 127.0.97.1:60247)
0923 19:30:01.427977 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:01.428042 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:01.430862 (+ 2820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:01.431834 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:01.431839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:01.432798 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:01.434394 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:01.434426 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.436077 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.436082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:01.438781 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:01.438797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.440163 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.440173 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:01.440370 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:01.441385 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:01.441416 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:01.445909 (+ 4493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:01.450718 (+ 4809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:01.450729 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:01.450755 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:01.451555 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:01.451983 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:01.451987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:01.451990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:01.452288 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:01.453020 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:01.453030 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:01.453634 (+ 604us) client_negotiation.cc:769] Sending connection context
0923 19:30:01.455164 (+ 1530us) client_negotiation.cc:240] Negotiation successful
0923 19:30:01.455497 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":258,"spinlock_wait_cycles":10240,"thread_start_us":452,"threads_started":1}
I20260923 19:30:01.457419 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:01.426138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39687 (local address 127.0.97.61:35879)
0923 19:30:01.426630 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:01.426635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:01.428517 (+ 1882us) server_negotiation.cc:407] Connection header received
0923 19:30:01.431205 (+ 2688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:01.431211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:01.431299 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:01.431378 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:01.440630 (+ 9252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.441834 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.444292 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.444969 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.445114 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:01.449606 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:01.449614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:01.449618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:01.449663 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:01.453117 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:01.454047 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:01.454052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:01.454056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:01.454194 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:01.454735 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:01.454740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:01.454744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:01.455514 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:01.455659 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:01.456259 (+ 600us) server_negotiation.cc:299] Negotiation successful
0923 19:30:01.456481 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":267,"threads_started":1}
Sep 23 19:30:01 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260923 19:30:01.458379 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:01.429039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33581 (local address 127.0.97.62:37425)
0923 19:30:01.430660 (+ 1621us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:01.430665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:01.430685 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:30:01.431185 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:01.431191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:01.431284 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:01.431395 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:01.435408 (+ 4013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.436664 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.438989 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.439636 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.440863 (+ 1227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:01.445508 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:01.445516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:01.445521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:01.445565 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:01.450787 (+ 5222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:01.451971 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:01.451976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:01.451980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:01.452127 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:01.452537 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:01.452541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:01.452544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:01.452878 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:01.457416 (+ 4538us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:01.457577 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 19:30:01.457947 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1510,"thread_start_us":1404,"threads_started":1}
I20260923 19:30:01.458644 7509 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44187
I20260923 19:30:01.459048 7509 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:01.459256 7510 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37425
I20260923 19:30:01.459399 7510 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:01.459264 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:01.427666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.97.1:39687)
0923 19:30:01.428423 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:01.428440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:01.430862 (+ 2422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:01.432912 (+ 2050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:01.432916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:01.439791 (+ 6875us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:01.440334 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:01.440356 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.442077 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.442084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:01.444065 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:01.444083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.445206 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.445213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:01.445430 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:01.446152 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:01.446175 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:01.449346 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:01.453287 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:01.453292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:01.453296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:01.453887 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:01.454418 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:01.454423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:01.454426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:01.454605 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:01.455749 (+ 1144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:01.455754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:01.455961 (+ 207us) client_negotiation.cc:769] Sending connection context
0923 19:30:01.458027 (+ 2066us) client_negotiation.cc:240] Negotiation successful
0923 19:30:01.458395 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":34,"spinlock_wait_cycles":2816,"thread_start_us":381,"threads_started":1}
I20260923 19:30:01.460220 7509 heartbeater.cc:511] Master 127.0.97.60:44187 requested a full tablet report, sending...
I20260923 19:30:01.460294 7510 heartbeater.cc:511] Master 127.0.97.62:37425 requested a full tablet report, sending...
I20260923 19:30:01.461481 7511 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35879
I20260923 19:30:01.461638 7511 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:01.462127 7511 heartbeater.cc:511] Master 127.0.97.61:35879 requested a full tablet report, sending...
I20260923 19:30:01.462947 6534 ts_manager.cc:194] Re-registered known tserver with Master: 066538205b95484ca562199aa7740b8a (127.0.97.1:43027)
I20260923 19:30:01.463780 7140 ts_manager.cc:194] Registered new tserver with Master: 066538205b95484ca562199aa7740b8a (127.0.97.1:43027)
I20260923 19:30:01.463765 7330 ts_manager.cc:194] Registered new tserver with Master: 066538205b95484ca562199aa7740b8a (127.0.97.1:43027)
I20260923 19:30:01.464097 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33581
I20260923 19:30:01.467662 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:01.433495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.0.97.2:37959)
0923 19:30:01.444836 (+ 11341us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:01.444843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:01.444869 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:30:01.444933 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:01.444940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:01.445217 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:01.445433 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:01.447756 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.449641 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.451574 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:01.452565 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:01.452695 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:01.458681 (+ 5986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:01.458696 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:01.458717 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:01.458763 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:01.463333 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:01.464794 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:01.464803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:01.464808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:01.464991 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:01.465569 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:01.465577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:01.465581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:01.466110 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:01.466516 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:01.466853 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 19:30:01.467156 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11158,"thread_start_us":10983,"threads_started":1}
I20260923 19:30:01.469655 6852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:01.593808 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: true } } }
I20260923 19:30:01.594220 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:01.599221 7262 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: attempting to promote NON_VOTER 0d25d7bc72b14cac94ddb530ede55d42 to VOTER
I20260923 19:30:01.600021 7262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } }
I20260923 19:30:01.602602 7340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 LEARNER]: Refusing update from remote peer 164e149b0070471e81a429e1c5fe1947: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 19:30:01.602653 7150 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Refusing update from remote peer 164e149b0070471e81a429e1c5fe1947: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 19:30:01.603266 7521 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 19:30:01.603767 7522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 19:30:01.613147 7262 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0d25d7bc72b14cac94ddb530ede55d42 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:01.617269 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:01.617571 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:01.618728 7150 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0d25d7bc72b14cac94ddb530ede55d42 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:01.624173 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:01.624593 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:01.633637 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:01.633952 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:01.634594 7525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:01.634208 7339 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0d25d7bc72b14cac94ddb530ede55d42 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:01.635015 7525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:30:01.639199 7368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "164e149b0070471e81a429e1c5fe1947" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:01.639520 7368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:30:01.648211 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 6729
I20260923 19:30:01.663105 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37959
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37425
--webserver_port=41169
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37425,127.0.97.61:35879
--tserver_master_addrs=127.0.97.62:37425,127.0.97.61:35879,127.0.97.60:44187 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:02.041925 7526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:02.042235 7526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:02.042383 7526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:02.049722 7526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:02.049852 7526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:02.050000 7526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:30:02.059939 7526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41169
--tserver_master_addrs=127.0.97.62:37425,127.0.97.61:35879,127.0.97.60:44187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:02.061312 7526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:02.062651 7526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:02.076731 7532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:02.077009 7534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:02.076731 7531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:02.078065 7526 server_base.cc:1034] running on GCE node
I20260923 19:30:02.078800 7526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:02.079823 7526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:02.081043 7526 hybrid_clock.cc:648] HybridClock initialized: now 1790191802080972 us; error 66 us; skew 500 ppm
I20260923 19:30:02.081377 7526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:02 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:02.084755 7526 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 19:30:02.086557 7526 webserver.cc:460] Webserver started at http://127.0.97.2:41169/ using document root <none> and password file <none>
I20260923 19:30:02.087314 7526 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:02.087432 7526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:02.093366 7526 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:30:02.096606 7541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:02.098265 7526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 19:30:02.098474 7526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "aaad02bea99c40c8a91cbd47e320579a"
format_stamp: "Formatted at 2026-09-23 19:29:58 on dist-test-slave-kxfb"
server_key: "85573eb76ce8515321174f692b043a77"
server_key_iv: "0562ae69270fce6bed76e14c8db44d41"
server_key_version: "encryptionkey@0"
I20260923 19:30:02.099447 7526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:02.133358 7526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:02.138278 7526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:02.138571 7526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:02.139578 7526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:02.141058 7526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:02.141149 7526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:02.141237 7526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:02.141292 7526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:02.174729 7526 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37959
I20260923 19:30:02.174858 7653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37959 every 8 connection(s)
I20260923 19:30:02.176245 7526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:30:02.182783 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 7526
I20260923 19:30:02.183112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 6863
Sep 23 19:30:02 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 23 19:30:02 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 23 19:30:02 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 19:30:02 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:30:02.243398 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.184372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.0.97.3:37351)
0923 19:30:02.195396 (+ 11024us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.195407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.217007 (+ 21600us) server_negotiation.cc:407] Connection header received
0923 19:30:02.217133 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.217151 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.217600 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.217816 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.222429 (+ 4613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.223986 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.225759 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.226615 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.226714 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.234320 (+ 7606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.234522 (+ 202us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.234546 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.234610 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.239304 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.240250 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.240259 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.240263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.240418 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.240852 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.240857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.240861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.241269 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.241542 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.242723 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.243020 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10827,"thread_start_us":10665,"threads_started":1}
I20260923 19:30:02.244225 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.200943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56723 (local address 127.0.97.60:44187)
0923 19:30:02.216681 (+ 15738us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.216689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.217402 (+ 713us) server_negotiation.cc:407] Connection header received
0923 19:30:02.217465 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.217470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.217551 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.217642 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.220326 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.221512 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.224929 (+ 3417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.225660 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.225769 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.233214 (+ 7445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.233222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.233227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.233275 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.238191 (+ 4916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.239321 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.239327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.239331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.239480 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.240103 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.240109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.240112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.240531 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.240678 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.242492 (+ 1814us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.242753 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15618,"thread_start_us":15800,"threads_started":1}
I20260923 19:30:02.245119 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.180377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.97.2:47009)
0923 19:30:02.215029 (+ 34652us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:02.215057 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:02.215234 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:02.227612 (+ 12378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:02.227619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:02.228494 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:02.229043 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.229064 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.230434 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.230440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.232059 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.232078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.233081 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.233087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.233485 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.234173 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:02.234196 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:02.236892 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:02.241497 (+ 4605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.241502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.241507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.242112 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.242615 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.242621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.242624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.242811 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.243449 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:02.243455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:02.243609 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 19:30:02.243766 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 19:30:02.244089 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34451,"mutex_wait_us":84,"spinlock_wait_cycles":37632,"thread_start_us":38954,"threads_started":1}
I20260923 19:30:02.245249 6986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:02.245532 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.207447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44187 (local address 127.0.97.2:56723)
0923 19:30:02.217156 (+ 9709us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:02.217174 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:02.217306 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:02.218157 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:02.218172 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:02.219233 (+ 1061us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:02.220138 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.220171 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.222492 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.222497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.224723 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.224739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.226168 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.226175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.226557 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.227437 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:02.227466 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:02.232882 (+ 5416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:02.238427 (+ 5545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.238438 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.238464 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.239161 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.239612 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.239617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.239621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.239909 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.240754 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:02.240764 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:02.241262 (+ 498us) client_negotiation.cc:769] Sending connection context
0923 19:30:02.241621 (+ 359us) client_negotiation.cc:240] Negotiation successful
0923 19:30:02.242223 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9596,"thread_start_us":7850,"threads_started":1}
I20260923 19:30:02.245898 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.179695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47009 (local address 127.0.97.61:35879)
0923 19:30:02.226430 (+ 46735us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.226438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.226472 (+ 34us) server_negotiation.cc:407] Connection header received
0923 19:30:02.226558 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.226565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.226678 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.227699 (+ 1021us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.229247 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.230186 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.232286 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.232862 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.233812 (+ 950us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.237211 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.237219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.237224 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.237267 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.241316 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.242326 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.242331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.242335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.242497 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.242957 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.242962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.242965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.243329 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.245095 (+ 1766us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.245230 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.245469 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46521,"thread_start_us":47671,"threads_started":1}
I20260923 19:30:02.246934 7654 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44187
I20260923 19:30:02.247371 7654 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:02.247380 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.180377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.2:33355)
0923 19:30:02.196126 (+ 15749us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:02.196193 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:02.198991 (+ 2798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:02.227613 (+ 28622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:02.227619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:02.228525 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:02.230022 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.230037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.231413 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.231419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.232551 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.232564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.233871 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.233877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.234058 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.234705 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:02.234727 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:02.238280 (+ 3553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:02.242619 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.242623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.242627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.243149 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.243574 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.243580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.243583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.243759 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.244437 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:02.244442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:02.244571 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 19:30:02.246604 (+ 2033us) client_negotiation.cc:240] Negotiation successful
0923 19:30:02.246982 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15366,"mutex_wait_us":40,"thread_start_us":19008,"threads_started":1}
I20260923 19:30:02.248162 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.179834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33355 (local address 127.0.97.62:37425)
0923 19:30:02.227266 (+ 47432us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.227272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.227293 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:30:02.227359 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.227364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.227442 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.227984 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.230247 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.231274 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.232693 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.233475 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.233608 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.238638 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.238647 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.238652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.238699 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.242446 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.243298 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.243303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.243306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.243463 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.243870 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.243875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.243878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.244223 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.244351 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.247547 (+ 3196us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.247819 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47288,"thread_start_us":47445,"threads_started":1}
I20260923 19:30:02.248468 7654 heartbeater.cc:511] Master 127.0.97.60:44187 requested a full tablet report, sending...
I20260923 19:30:02.249002 7655 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37425
I20260923 19:30:02.249164 7655 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:02.249500 7656 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35879
I20260923 19:30:02.249681 7656 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:02.249821 7655 heartbeater.cc:511] Master 127.0.97.62:37425 requested a full tablet report, sending...
I20260923 19:30:02.250681 7656 heartbeater.cc:511] Master 127.0.97.61:35879 requested a full tablet report, sending...
I20260923 19:30:02.251641 7330 ts_manager.cc:194] Registered new tserver with Master: aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959)
I20260923 19:30:02.252031 7140 ts_manager.cc:194] Registered new tserver with Master: aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959)
I20260923 19:30:02.252365 6534 ts_manager.cc:194] Re-registered known tserver with Master: aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959)
I20260923 19:30:02.253645 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33355
I20260923 19:30:02.340036 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 6863
W20260923 19:30:02.353214 6520 connection.cc:535] server connection from 127.0.97.3:38897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 19:30:02.353791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37351
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37425
--webserver_port=41895
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37425,127.0.97.61:35879
--tserver_master_addrs=127.0.97.62:37425,127.0.97.61:35879,127.0.97.60:44187 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:30:02.467440 7510 heartbeater.cc:503] Master 127.0.97.62:37425 was elected leader, sending a full tablet report...
W20260923 19:30:02.726624 7667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:02.727010 7667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:02.727147 7667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:02.735385 7667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:02.735535 7667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:02.735702 7667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:30:02.746783 7667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41895
--tserver_master_addrs=127.0.97.62:37425,127.0.97.61:35879,127.0.97.60:44187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:02.748495 7667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:02.750451 7667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:02.767349 7673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:02.768136 7672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:02.768329 7675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:02.769906 7667 server_base.cc:1034] running on GCE node
I20260923 19:30:02.770918 7667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:02.772555 7667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:02.773844 7667 hybrid_clock.cc:648] HybridClock initialized: now 1790191802773787 us; error 48 us; skew 500 ppm
I20260923 19:30:02.774353 7667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:02 dist-test-slave-kxfb krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:02.779028 7667 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 19:30:02.781118 7667 webserver.cc:460] Webserver started at http://127.0.97.3:41895/ using document root <none> and password file <none>
I20260923 19:30:02.782209 7667 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:02.782405 7667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:02.790735 7667 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 19:30:02.794826 7682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:02.796872 7667 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:30:02.797083 7667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "130dae76d5ba48beb6873ed7e28d8d13"
format_stamp: "Formatted at 2026-09-23 19:29:59 on dist-test-slave-kxfb"
server_key: "cc90d8bc43613603c88266e2c96f78d5"
server_key_iv: "fcf34b8bed4cc906e6a19ab4de85a582"
server_key_version: "encryptionkey@0"
I20260923 19:30:02.798676 7667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:02.818113 7667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:02.824252 7667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:02.824637 7667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:02.826064 7667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:02.828161 7667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:02.828295 7667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:02.828438 7667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:02.828522 7667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:02.868476 7667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37351
I20260923 19:30:02.868657 7794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37351 every 8 connection(s)
I20260923 19:30:02.870754 7667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:30:02.877767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 7667
Sep 23 19:30:02 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 19:30:02 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 23 19:30:02 dist-test-slave-kxfb krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 19:30:02.930389 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.881503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.97.3:39107)
0923 19:30:02.890839 (+ 9336us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:02.890866 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:02.897780 (+ 6914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:02.898351 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:02.898356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:02.899312 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:02.900563 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.900601 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.904467 (+ 3866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.904476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.909708 (+ 5232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.909723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.912166 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.912173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.912486 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.913400 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:02.913433 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:02.916733 (+ 3300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:02.921214 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.921225 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.921245 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.921936 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.923533 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.923539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.923543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.923802 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.927344 (+ 3542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:02.927355 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:02.927903 (+ 548us) client_negotiation.cc:769] Sending connection context
0923 19:30:02.928256 (+ 353us) client_negotiation.cc:240] Negotiation successful
0923 19:30:02.928866 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9174,"thread_start_us":4953,"threads_started":1}
I20260923 19:30:02.934846 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.882109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39107 (local address 127.0.97.61:35879)
0923 19:30:02.890431 (+ 8322us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.890440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.890970 (+ 530us) server_negotiation.cc:407] Connection header received
0923 19:30:02.898003 (+ 7033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.898011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.898134 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.898303 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.900814 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.902091 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.909858 (+ 7767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.910610 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.910754 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.917053 (+ 6299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.917061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.917067 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.917116 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.921076 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.922065 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.922071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.922075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.922686 (+ 611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.925224 (+ 2538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.925232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.925236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.925682 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.933964 (+ 8282us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.934130 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.934456 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8138,"thread_start_us":8021,"threads_started":1}
I20260923 19:30:02.936386 7797 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35879
I20260923 19:30:02.936764 7797 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:02.937747 7797 heartbeater.cc:511] Master 127.0.97.61:35879 requested a full tablet report, sending...
I20260923 19:30:02.939849 7140 ts_manager.cc:194] Registered new tserver with Master: 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351)
I20260923 19:30:02.950492 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.873971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37209 (local address 127.0.97.62:37425)
0923 19:30:02.877614 (+ 3643us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.877622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.887414 (+ 9792us) server_negotiation.cc:407] Connection header received
0923 19:30:02.895964 (+ 8550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.895971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.896094 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.896253 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.902853 (+ 6600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.904048 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.917052 (+ 13004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.917736 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.917848 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.939158 (+ 21310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.939167 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.939173 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.939226 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.943114 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.945782 (+ 2668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.945789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.945793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.945967 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.948519 (+ 2552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.948525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.948529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.948981 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.949142 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.949709 (+ 567us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.949999 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3448,"thread_start_us":3267,"threads_started":1}
I20260923 19:30:02.951845 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.874747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.3:37209)
0923 19:30:02.887032 (+ 12285us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:02.887092 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:02.895429 (+ 8337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:02.897111 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:02.897129 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:02.900699 (+ 3570us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:02.902638 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.902653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.904374 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.904381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.908043 (+ 3662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.908074 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.918171 (+ 10097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.918178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.918391 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.922173 (+ 3782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:02.922198 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:02.933688 (+ 11490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:02.944344 (+ 10656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.944350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.944354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.944892 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.946120 (+ 1228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.946124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.946128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.946353 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.949234 (+ 2881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:02.949239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:02.949397 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 19:30:02.951097 (+ 1700us) client_negotiation.cc:240] Negotiation successful
0923 19:30:02.951423 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11862,"thread_start_us":15592,"threads_started":1}
I20260923 19:30:02.953624 7796 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37425
I20260923 19:30:02.953778 7796 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:02.954257 7796 heartbeater.cc:511] Master 127.0.97.62:37425 requested a full tablet report, sending...
I20260923 19:30:02.955513 6534 ts_manager.cc:194] Re-registered known tserver with Master: 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351)
I20260923 19:30:02.956773 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37209
I20260923 19:30:02.960151 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.891426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44187 (local address 127.0.97.3:51883)
0923 19:30:02.908658 (+ 17232us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:02.908681 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:02.908829 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:02.910410 (+ 1581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:02.910416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:02.911567 (+ 1151us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:02.911871 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.911883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.919044 (+ 7161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.919050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.920547 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:02.920562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.924672 (+ 4110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.924679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:02.924870 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.925894 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:02.925916 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:02.931650 (+ 5734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:02.953087 (+ 21437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.953092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.953097 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.955061 (+ 1964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.955594 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:02.955599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:02.955603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:02.955778 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:02.956477 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:02.956483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:02.956635 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 19:30:02.956753 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 19:30:02.957078 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17070,"thread_start_us":14436,"threads_started":1}
I20260923 19:30:02.961483 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.891415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51883 (local address 127.0.97.60:44187)
0923 19:30:02.905714 (+ 14299us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.905723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.909877 (+ 4154us) server_negotiation.cc:407] Connection header received
0923 19:30:02.910006 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.910012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.910098 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.910196 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.913008 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.918851 (+ 5843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.920729 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.924243 (+ 3514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.924368 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.931948 (+ 7580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.931955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.931961 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.932006 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.948345 (+ 16339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.955257 (+ 6912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.955264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.955268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.955456 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.955933 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.955938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.955942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.956334 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.960581 (+ 4247us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.960746 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.961062 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12749,"thread_start_us":12786,"threads_started":1}
I20260923 19:30:02.962440 7795 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44187
I20260923 19:30:02.962596 7795 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:02.963040 7795 heartbeater.cc:511] Master 127.0.97.60:44187 requested a full tablet report, sending...
I20260923 19:30:02.963977 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.908136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.0.97.61:35879)
0923 19:30:02.933729 (+ 25593us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.933737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.933773 (+ 36us) server_negotiation.cc:407] Connection header received
0923 19:30:02.933839 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.933845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.933938 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.934060 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.938662 (+ 4602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.939848 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.941820 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.942454 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.942572 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.952108 (+ 9536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.952115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.952119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.952158 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.957085 (+ 4927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.958270 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.958276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.958280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.958456 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.958875 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.958880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.958883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.959264 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.959498 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.960668 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.960915 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25435,"thread_start_us":27067,"threads_started":1}
I20260923 19:30:02.964413 7330 ts_manager.cc:194] Registered new tserver with Master: 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351)
I20260923 19:30:02.971509 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.928364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.0.97.60:44187)
0923 19:30:02.942115 (+ 13751us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.942122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.944087 (+ 1965us) server_negotiation.cc:407] Connection header received
0923 19:30:02.944225 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.944231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.944315 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.944439 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.947005 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.948403 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.950239 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.950962 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.951081 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.962501 (+ 11420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.962510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.962574 (+ 64us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.962629 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.967236 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.968293 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.968299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.968303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.968532 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.969031 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.969036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.969040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.969418 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.969633 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.970392 (+ 759us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.971077 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13602,"thread_start_us":13772,"threads_started":1}
I20260923 19:30:02.972735 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:02.908123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.0.97.62:37425)
0923 19:30:02.937667 (+ 29544us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:02.937675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:02.937699 (+ 24us) server_negotiation.cc:407] Connection header received
0923 19:30:02.937770 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:02.937776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:02.937996 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:02.938118 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:02.948821 (+ 10703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.950108 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.957782 (+ 7674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:02.958451 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:02.958547 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:02.965415 (+ 6868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:02.965422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:02.965426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:02.965465 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:02.969245 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.970254 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.970259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.970262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.970447 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:02.970918 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:02.970923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:02.970926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:02.971285 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:02.971516 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:02.972122 (+ 606us) server_negotiation.cc:299] Negotiation successful
0923 19:30:02.972370 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29388,"thread_start_us":28574,"threads_started":1}
I20260923 19:30:02.977389 6534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:30:02.985075 6534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:30:03.056921 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.009156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43027 (local address 127.0.0.1:57752)
0923 19:30:03.046364 (+ 37208us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:03.046392 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:03.046517 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:03.047605 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:03.047610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:03.047636 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:03.048009 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.048021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.050396 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.050403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.053783 (+ 3380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.053809 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.056067 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.056073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.056255 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.056292 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 19:30:03.056380 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 19:30:03.056544 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37062,"thread_start_us":36949,"threads_started":1}
I20260923 19:30:03.057660 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.009727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57752 (local address 127.0.97.1:43027)
0923 19:30:03.043067 (+ 33340us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.043081 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.046492 (+ 3411us) server_negotiation.cc:407] Connection header received
0923 19:30:03.046684 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.046696 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.047053 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.047346 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:03.048280 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.050032 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.054028 (+ 3996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.055535 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.055671 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.055891 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.057184 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.057368 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33025,"thread_start_us":33387,"threads_started":1}
I20260923 19:30:03.059597 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.009552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37959 (local address 127.0.0.1:35154)
0923 19:30:03.048921 (+ 39369us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:03.048940 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:03.049051 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:03.049924 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:03.049929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:03.049951 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:03.050397 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.050408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.052484 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.052488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.056116 (+ 3628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.056130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.058702 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.058720 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.058886 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.058920 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:30:03.058999 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 19:30:03.059152 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39264,"mutex_wait_us":1,"thread_start_us":35546,"threads_started":1}
I20260923 19:30:03.060506 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.009834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.0.97.2:37959)
0923 19:30:03.047055 (+ 37221us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.047064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.049173 (+ 2109us) server_negotiation.cc:407] Connection header received
0923 19:30:03.049282 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.049288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.049514 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.049694 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:03.051386 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.052361 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.056297 (+ 3936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.058156 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.058261 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.058518 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.060023 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.060214 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37037,"thread_start_us":36933,"threads_started":1}
I20260923 19:30:03.063948 7589 tablet_service.cc:1467] Processing CreateTablet for tablet 57b49781466b4fc1ad7c0f7a73cab45b (DEFAULT_TABLE table=test-table [id=2f941e85d9544e7f98fba88a76050df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:03.063948 7444 tablet_service.cc:1467] Processing CreateTablet for tablet 57b49781466b4fc1ad7c0f7a73cab45b (DEFAULT_TABLE table=test-table [id=2f941e85d9544e7f98fba88a76050df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:03.065215 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.009216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.0.97.3:37351)
0923 19:30:03.047662 (+ 38446us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.047670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.047723 (+ 53us) server_negotiation.cc:407] Connection header received
0923 19:30:03.047825 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.047832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.048068 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.048253 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:03.048866 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.050532 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.057195 (+ 6663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.062469 (+ 5274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.062609 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.062858 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.064474 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.064675 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38246,"thread_start_us":38041,"threads_started":1}
I20260923 19:30:03.065650 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.008802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37351 (local address 127.0.0.1:54046)
0923 19:30:03.045402 (+ 36600us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:03.045423 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:03.045627 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:03.048244 (+ 2617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:03.048248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:03.048272 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:03.048588 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.048602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.050680 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.050685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.056999 (+ 6314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.057013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.063055 (+ 6042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.063061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.063259 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.063295 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 19:30:03.063381 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 19:30:03.063538 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36443,"mutex_wait_us":21,"thread_start_us":41906,"threads_started":1}
I20260923 19:30:03.065656 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b49781466b4fc1ad7c0f7a73cab45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:03.065697 7444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b49781466b4fc1ad7c0f7a73cab45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:03.071326 7730 tablet_service.cc:1467] Processing CreateTablet for tablet 57b49781466b4fc1ad7c0f7a73cab45b (DEFAULT_TABLE table=test-table [id=2f941e85d9544e7f98fba88a76050df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:03.073665 7730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b49781466b4fc1ad7c0f7a73cab45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:03.083093 7831 tablet_bootstrap.cc:492] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: Bootstrap starting.
I20260923 19:30:03.086611 7831 tablet_bootstrap.cc:654] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:03.087158 7832 tablet_bootstrap.cc:492] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: Bootstrap starting.
I20260923 19:30:03.089327 7831 log.cc:824] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:03.091261 7832 tablet_bootstrap.cc:654] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:03.092458 7832 log.cc:824] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:03.092967 7831 tablet_bootstrap.cc:492] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: No bootstrap required, opened a new log
I20260923 19:30:03.093273 7831 ts_tablet_manager.cc:1397] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 19:30:03.093518 7834 tablet_bootstrap.cc:492] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: Bootstrap starting.
I20260923 19:30:03.097195 7834 tablet_bootstrap.cc:654] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:03.098438 7834 log.cc:824] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:03.099469 7832 tablet_bootstrap.cc:492] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: No bootstrap required, opened a new log
I20260923 19:30:03.099838 7832 ts_tablet_manager.cc:1397] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260923 19:30:03.103700 7834 tablet_bootstrap.cc:492] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: No bootstrap required, opened a new log
I20260923 19:30:03.103606 7831 raft_consensus.cc:348] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:03.103998 7834 ts_tablet_manager.cc:1397] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 19:30:03.104063 7831 raft_consensus.cc:374] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:03.104146 7831 raft_consensus.cc:729] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066538205b95484ca562199aa7740b8a, State: Initialized, Role: FOLLOWER
I20260923 19:30:03.104925 7831 consensus_queue.cc:260] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:03.109681 7832 raft_consensus.cc:348] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:03.110131 7832 raft_consensus.cc:374] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:03.110257 7832 raft_consensus.cc:729] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaad02bea99c40c8a91cbd47e320579a, State: Initialized, Role: FOLLOWER
I20260923 19:30:03.111778 7831 ts_tablet_manager.cc:1428] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260923 19:30:03.111635 7832 consensus_queue.cc:260] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:03.116775 7834 raft_consensus.cc:348] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:03.117255 7834 raft_consensus.cc:374] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:03.117348 7834 raft_consensus.cc:729] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130dae76d5ba48beb6873ed7e28d8d13, State: Initialized, Role: FOLLOWER
I20260923 19:30:03.118076 7834 consensus_queue.cc:260] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:03.121417 7834 ts_tablet_manager.cc:1428] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260923 19:30:03.121740 7796 heartbeater.cc:503] Master 127.0.97.62:37425 was elected leader, sending a full tablet report...
I20260923 19:30:03.123934 7832 ts_tablet_manager.cc:1428] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: Time spent starting tablet: real 0.024s user 0.010s sys 0.013s
I20260923 19:30:03.124276 7655 heartbeater.cc:503] Master 127.0.97.62:37425 was elected leader, sending a full tablet report...
W20260923 19:30:03.138623 7801 tablet.cc:2369] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:30:03.145277 7837 raft_consensus.cc:482] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:03.145561 7837 raft_consensus.cc:504] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:03.147270 7837 leader_election.cc:288] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351), aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959)
I20260923 19:30:03.154906 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.147654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56541 (local address 127.0.97.3:37351)
0923 19:30:03.147850 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.147857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.148681 (+ 824us) server_negotiation.cc:407] Connection header received
0923 19:30:03.148876 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.148881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.148942 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.149021 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:03.149613 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.150556 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.152988 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.153760 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.153864 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.153969 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.154587 (+ 618us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.154692 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 19:30:03.155099 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.147602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37351 (local address 127.0.97.1:56541)
0923 19:30:03.148558 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:03.148579 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:03.148743 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:03.149051 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:03.149055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:03.149087 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:03.149468 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.149484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.151192 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.151196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.152830 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.152844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.154089 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.154099 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.154343 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.154383 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 19:30:03.154490 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 19:30:03.154641 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":24,"thread_start_us":828,"threads_started":1}
I20260923 19:30:03.155829 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.147786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57841 (local address 127.0.97.2:37959)
0923 19:30:03.148028 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.148036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.148866 (+ 830us) server_negotiation.cc:407] Connection header received
0923 19:30:03.149059 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.149064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.149128 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.149205 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:03.149821 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.150737 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.152810 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.154636 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.154785 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.154888 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.155433 (+ 545us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.155567 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 19:30:03.155881 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.147640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37959 (local address 127.0.97.1:57841)
0923 19:30:03.148748 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:03.148769 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:03.148925 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:03.149238 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:03.149242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:03.149269 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:03.149654 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.149663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.150887 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.150893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.152644 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:03.152657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.155045 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.155051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:03.155232 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.155263 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 19:30:03.155388 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 19:30:03.155533 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":3,"thread_start_us":427,"threads_started":1}
I20260923 19:30:03.156256 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b49781466b4fc1ad7c0f7a73cab45b" candidate_uuid: "066538205b95484ca562199aa7740b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130dae76d5ba48beb6873ed7e28d8d13" is_pre_election: true
I20260923 19:30:03.156737 7750 raft_consensus.cc:2454] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 066538205b95484ca562199aa7740b8a in term 0.
I20260923 19:30:03.156894 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b49781466b4fc1ad7c0f7a73cab45b" candidate_uuid: "066538205b95484ca562199aa7740b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaad02bea99c40c8a91cbd47e320579a" is_pre_election: true
I20260923 19:30:03.157369 7398 leader_election.cc:302] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066538205b95484ca562199aa7740b8a, 130dae76d5ba48beb6873ed7e28d8d13; no voters:
I20260923 19:30:03.157419 7609 raft_consensus.cc:2454] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 066538205b95484ca562199aa7740b8a in term 0.
I20260923 19:30:03.157783 7837 raft_consensus.cc:2781] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:30:03.157930 7837 raft_consensus.cc:482] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:30:03.158039 7837 raft_consensus.cc:3037] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:03.160465 7837 raft_consensus.cc:504] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:03.161164 7837 leader_election.cc:288] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [CANDIDATE]: Term 1 election: Requested vote from peers 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351), aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959)
I20260923 19:30:03.161747 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b49781466b4fc1ad7c0f7a73cab45b" candidate_uuid: "066538205b95484ca562199aa7740b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130dae76d5ba48beb6873ed7e28d8d13"
I20260923 19:30:03.161945 7750 raft_consensus.cc:3037] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:03.161882 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b49781466b4fc1ad7c0f7a73cab45b" candidate_uuid: "066538205b95484ca562199aa7740b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaad02bea99c40c8a91cbd47e320579a"
I20260923 19:30:03.162138 7609 raft_consensus.cc:3037] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:03.164364 7750 raft_consensus.cc:2454] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 066538205b95484ca562199aa7740b8a in term 1.
I20260923 19:30:03.164610 7609 raft_consensus.cc:2454] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 066538205b95484ca562199aa7740b8a in term 1.
I20260923 19:30:03.164810 7398 leader_election.cc:302] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066538205b95484ca562199aa7740b8a, 130dae76d5ba48beb6873ed7e28d8d13; no voters:
I20260923 19:30:03.165190 7837 raft_consensus.cc:2781] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:03.165652 7837 raft_consensus.cc:686] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 1 LEADER]: Becoming Leader. State: Replica: 066538205b95484ca562199aa7740b8a, State: Running, Role: LEADER
I20260923 19:30:03.166324 7837 consensus_queue.cc:237] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:03.172106 6533 catalog_manager.cc:5495] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 066538205b95484ca562199aa7740b8a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "066538205b95484ca562199aa7740b8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } health_report { overall_health: UNKNOWN } } }
W20260923 19:30:03.179980 7512 tablet.cc:2369] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:30:03.180217 7657 tablet.cc:2369] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:30:03.220184 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.214073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.0.97.1:43027)
0923 19:30:03.214355 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.214362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.214386 (+ 24us) server_negotiation.cc:407] Connection header received
0923 19:30:03.214726 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.214733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.214848 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.215030 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:03.215417 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.216257 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.217688 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.218151 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.218257 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.218874 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:03.219582 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:03.219726 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.219804 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.219881 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 19:30:03.229888 7609 raft_consensus.cc:1261] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [term 1 FOLLOWER]: Refusing update from remote peer 066538205b95484ca562199aa7740b8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:03.229966 7750 raft_consensus.cc:1261] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 066538205b95484ca562199aa7740b8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:03.231076 7837 consensus_queue.cc:1035] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:03.231407 7842 consensus_queue.cc:1035] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:03.250008 7848 mvcc.cc:205] Tried to move back new op lower bound from 7332625626017701888 to 7332625625773588480. Current Snapshot: MvccSnapshot[applied={T|T < 7332625626017701888}]
I20260923 19:30:03.266028 6533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:03.266391 6533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:03.274055 6533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 164e149b0070471e81a429e1c5fe1947: Sending DeleteTablet for 3 replicas of tablet 57b49781466b4fc1ad7c0f7a73cab45b
I20260923 19:30:03.275421 7730 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b49781466b4fc1ad7c0f7a73cab45b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54046
I20260923 19:30:03.276093 7444 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b49781466b4fc1ad7c0f7a73cab45b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57752
I20260923 19:30:03.276546 7855 tablet_replica.cc:330] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: stopping tablet replica
I20260923 19:30:03.276940 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b49781466b4fc1ad7c0f7a73cab45b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35154
I20260923 19:30:03.277025 7855 raft_consensus.cc:2229] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:03.277520 7855 raft_consensus.cc:2258] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:03.279294 7856 tablet_replica.cc:330] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: stopping tablet replica
I20260923 19:30:03.279961 7856 raft_consensus.cc:2229] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:30:03.280866 7856 raft_consensus.cc:2258] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:03.284142 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 6502
I20260923 19:30:03.284222 7857 tablet_replica.cc:330] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: stopping tablet replica
I20260923 19:30:03.285357 6564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:03.285655 7855 ts_tablet_manager.cc:1905] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:03.286250 7856 ts_tablet_manager.cc:1905] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:03.292837 7855 ts_tablet_manager.cc:1918] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:03.293107 7855 log.cc:1196] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/57b49781466b4fc1ad7c0f7a73cab45b
I20260923 19:30:03.293668 7855 ts_tablet_manager.cc:1939] T 57b49781466b4fc1ad7c0f7a73cab45b P 130dae76d5ba48beb6873ed7e28d8d13: Deleting consensus metadata
I20260923 19:30:03.294071 7857 raft_consensus.cc:2229] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:03.294646 7857 raft_consensus.cc:2258] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:03.300418 7857 ts_tablet_manager.cc:1905] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:03.308267 7857 ts_tablet_manager.cc:1918] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:03.308466 7857 log.cc:1196] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/57b49781466b4fc1ad7c0f7a73cab45b
I20260923 19:30:03.308691 7856 ts_tablet_manager.cc:1918] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:03.308940 7856 log.cc:1196] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/57b49781466b4fc1ad7c0f7a73cab45b
I20260923 19:30:03.309156 7857 ts_tablet_manager.cc:1939] T 57b49781466b4fc1ad7c0f7a73cab45b P aaad02bea99c40c8a91cbd47e320579a: Deleting consensus metadata
I20260923 19:30:03.309640 7856 ts_tablet_manager.cc:1939] T 57b49781466b4fc1ad7c0f7a73cab45b P 066538205b95484ca562199aa7740b8a: Deleting consensus metadata
I20260923 19:30:03.318674 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.278941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.0.97.61:35879)
0923 19:30:03.279138 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.279145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.290141 (+ 10996us) server_negotiation.cc:407] Connection header received
0923 19:30:03.290542 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.290548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.290668 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.290754 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:03.295445 (+ 4691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.296728 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.302698 (+ 5970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.303513 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.303670 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.311893 (+ 8223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:03.311902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:03.311909 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:03.311974 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:03.315711 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:03.316613 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:03.316620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:03.316623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:03.316771 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:03.317140 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:03.317145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:03.317148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:03.317467 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:03.317716 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.318055 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.318326 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 19:30:03.332674 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.283037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.0.97.60:44187)
0923 19:30:03.283221 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.283228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.318932 (+ 35704us) server_negotiation.cc:407] Connection header received
0923 19:30:03.319190 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.319196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.319323 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.319397 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:03.321911 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.322729 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.324127 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.324670 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.324779 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.327341 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:03.327347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:03.327350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:03.327382 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:03.330058 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:03.330796 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:03.330809 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:03.330813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:03.330928 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:03.331276 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:03.331281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:03.331285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:03.331562 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:03.331808 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.332120 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.332341 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 19:30:03.488745 6520 catalog_manager.cc:4841] TS aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959): tablet 57b49781466b4fc1ad7c0f7a73cab45b (table test-table [id=2f941e85d9544e7f98fba88a76050df5]) successfully deleted
I20260923 19:30:03.490368 6519 catalog_manager.cc:4841] TS 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351): tablet 57b49781466b4fc1ad7c0f7a73cab45b (table test-table [id=2f941e85d9544e7f98fba88a76050df5]) successfully deleted
W20260923 19:30:03.490669 6519 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.3:37351, user_credentials={real_user=slave}} blocked reactor thread for 194959us
I20260923 19:30:03.490809 6521 catalog_manager.cc:4841] TS 066538205b95484ca562199aa7740b8a (127.0.97.1:43027): tablet 57b49781466b4fc1ad7c0f7a73cab45b (table test-table [id=2f941e85d9544e7f98fba88a76050df5]) successfully deleted
I20260923 19:30:03.491809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 6502
I20260923 19:30:03.540235 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.518835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.0.97.61:35879)
0923 19:30:03.519083 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.519090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.519993 (+ 903us) server_negotiation.cc:407] Connection header received
0923 19:30:03.520474 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.520481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.520580 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.520660 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:03.523216 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.524516 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.527686 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.528375 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.528479 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.532981 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:03.532988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:03.532993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:03.533041 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:03.536913 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:03.538012 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:03.538018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:03.538023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:03.538186 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:03.538659 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:03.538664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:03.538667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:03.539062 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:03.539321 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.539647 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.539869 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 19:30:03.547591 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:03.520100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.0.97.60:44187)
0923 19:30:03.520272 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:03.520279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:03.529956 (+ 9677us) server_negotiation.cc:407] Connection header received
0923 19:30:03.530235 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:03.530241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:03.530369 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:03.530447 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:03.532750 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.535227 (+ 2477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.537665 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:03.538215 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:03.538396 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:03.542257 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:03.542262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:03.542266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:03.542339 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:03.544992 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:03.545611 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:03.545617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:03.545621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:03.545742 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:03.546081 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:03.546086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:03.546090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:03.546459 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:03.546677 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:03.547027 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 19:30:03.547241 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 19:30:04.258345 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.257445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.2:38659)
0923 19:30:04.257937 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.258103 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":236,"threads_started":1}
W20260923 19:30:04.258937 7655 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37425 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
I20260923 19:30:04.260470 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.259686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:39637)
0923 19:30:04.260152 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.260280 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":256,"threads_started":1}
W20260923 19:30:04.260875 7510 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37425 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
I20260923 19:30:04.262509 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.261515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.3:47513)
0923 19:30:04.262090 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.262251 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":325,"threads_started":1}
W20260923 19:30:04.262948 7796 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37425 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
I20260923 19:30:04.774586 7870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 164e149b0070471e81a429e1c5fe1947)
I20260923 19:30:04.774919 7871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 164e149b0070471e81a429e1c5fe1947)
I20260923 19:30:04.774922 7870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } }
I20260923 19:30:04.775149 7871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } }
I20260923 19:30:04.776099 7871 leader_election.cc:288] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425), 91b529dd056e4b7aa9933aa1c68d339f (127.0.97.61:35879)
I20260923 19:30:04.777617 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.776405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54184)
0923 19:30:04.777076 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.777310 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":313,"threads_started":1}
I20260923 19:30:04.777822 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.776538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54190)
0923 19:30:04.777313 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.777484 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":540,"threads_started":1}
W20260923 19:30:04.778065 7315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
W20260923 19:30:04.778610 7126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
I20260923 19:30:04.780500 7870 leader_election.cc:288] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425), 0d25d7bc72b14cac94ddb530ede55d42 (127.0.97.60:44187)
I20260923 19:30:04.780967 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.780093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54200)
0923 19:30:04.780564 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.780715 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":234,"threads_started":1}
W20260923 19:30:04.781190 7126 leader_election.cc:334] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
I20260923 19:30:04.783056 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.782061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54202)
0923 19:30:04.782616 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.782765 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437}
W20260923 19:30:04.783222 7315 leader_election.cc:334] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
I20260923 19:30:04.788429 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.779651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44187 (local address 127.0.0.1:46432)
0923 19:30:04.779974 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.779991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:04.780110 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:04.782046 (+ 1936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:04.782051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:04.782088 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:04.782403 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:04.782413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:04.783870 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:04.783875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:04.785791 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:04.785806 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:04.787700 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:04.787705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:04.787862 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:04.787898 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 19:30:04.788015 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 19:30:04.788160 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":91}
I20260923 19:30:04.788682 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.781059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.0.97.60:44187)
0923 19:30:04.781730 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:04.781734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:04.781751 (+ 17us) server_negotiation.cc:407] Connection header received
0923 19:30:04.781819 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:04.781824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:04.781891 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:04.782124 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:04.782574 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:04.783743 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:04.786008 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:04.787471 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:04.787640 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:04.787792 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:04.788074 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 19:30:04.788250 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":736,"threads_started":1}
I20260923 19:30:04.789240 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.777285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.0.97.61:35879)
0923 19:30:04.778621 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:04.778627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:04.781542 (+ 2915us) server_negotiation.cc:407] Connection header received
0923 19:30:04.781821 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:04.781826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:04.781891 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:04.781988 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:04.783131 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:04.784253 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:04.786442 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:04.787804 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:04.787937 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:04.788050 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:04.788668 (+ 618us) server_negotiation.cc:299] Negotiation successful
0923 19:30:04.788836 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1235,"spinlock_wait_cycles":7040,"thread_start_us":1092,"threads_started":1}
I20260923 19:30:04.789892 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.776417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35879 (local address 127.0.0.1:59326)
0923 19:30:04.781394 (+ 4977us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.781419 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:04.781650 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:04.782567 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:04.782572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:04.782602 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:04.782907 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:04.782921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:04.784416 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:04.784422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:04.786256 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:04.786267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:04.788201 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:04.788209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:04.788440 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:04.788483 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 19:30:04.789463 (+ 980us) client_negotiation.cc:240] Negotiation successful
0923 19:30:04.789604 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4871,"mutex_wait_us":38,"thread_start_us":5922,"threads_started":1}
I20260923 19:30:04.790236 7339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91b529dd056e4b7aa9933aa1c68d339f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d25d7bc72b14cac94ddb530ede55d42" is_pre_election: true
I20260923 19:30:04.790613 7339 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91b529dd056e4b7aa9933aa1c68d339f in term 1.
I20260923 19:30:04.790604 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d25d7bc72b14cac94ddb530ede55d42" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91b529dd056e4b7aa9933aa1c68d339f" is_pre_election: true
I20260923 19:30:04.790835 7150 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d25d7bc72b14cac94ddb530ede55d42 in term 1.
I20260923 19:30:04.791148 7126 leader_election.cc:302] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d25d7bc72b14cac94ddb530ede55d42, 91b529dd056e4b7aa9933aa1c68d339f; no voters: 164e149b0070471e81a429e1c5fe1947
I20260923 19:30:04.791489 7870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 19:30:04.791622 7870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 164e149b0070471e81a429e1c5fe1947)
I20260923 19:30:04.791788 7870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 1 FOLLOWER]: Advancing to term 2
I20260923 19:30:04.791786 7314 leader_election.cc:302] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d25d7bc72b14cac94ddb530ede55d42, 91b529dd056e4b7aa9933aa1c68d339f; no voters: 164e149b0070471e81a429e1c5fe1947
I20260923 19:30:04.792133 7871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 19:30:04.792279 7871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 164e149b0070471e81a429e1c5fe1947)
I20260923 19:30:04.792404 7871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 1 FOLLOWER]: Advancing to term 2
I20260923 19:30:04.795357 7870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } }
I20260923 19:30:04.795550 7871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } }
I20260923 19:30:04.796283 7870 leader_election.cc:288] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [CANDIDATE]: Term 2 election: Requested vote from peers 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425), 0d25d7bc72b14cac94ddb530ede55d42 (127.0.97.60:44187)
I20260923 19:30:04.796334 7871 leader_election.cc:288] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 2 election: Requested vote from peers 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425), 91b529dd056e4b7aa9933aa1c68d339f (127.0.97.61:35879)
I20260923 19:30:04.797256 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.796755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54232)
0923 19:30:04.796908 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.797017 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 19:30:04.797272 7339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91b529dd056e4b7aa9933aa1c68d339f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d25d7bc72b14cac94ddb530ede55d42"
I20260923 19:30:04.797340 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d25d7bc72b14cac94ddb530ede55d42" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91b529dd056e4b7aa9933aa1c68d339f"
I20260923 19:30:04.797577 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.796392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54230)
0923 19:30:04.797277 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.797383 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":752}
I20260923 19:30:04.797634 7339 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 91b529dd056e4b7aa9933aa1c68d339f in current term 2: Already voted for candidate 0d25d7bc72b14cac94ddb530ede55d42 in this term.
I20260923 19:30:04.797698 7150 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d25d7bc72b14cac94ddb530ede55d42 in current term 2: Already voted for candidate 91b529dd056e4b7aa9933aa1c68d339f in this term.
I20260923 19:30:04.798638 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.798233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54242)
0923 19:30:04.798432 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.798516 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 19:30:04.798820 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:04.798377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54258)
0923 19:30:04.798586 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:04.798690 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260923 19:30:04.798840 7126 leader_election.cc:334] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
W20260923 19:30:04.799013 7315 leader_election.cc:334] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
I20260923 19:30:04.799008 7126 leader_election.cc:302] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91b529dd056e4b7aa9933aa1c68d339f; no voters: 0d25d7bc72b14cac94ddb530ede55d42, 164e149b0070471e81a429e1c5fe1947
I20260923 19:30:04.799127 7315 leader_election.cc:302] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d25d7bc72b14cac94ddb530ede55d42; no voters: 164e149b0070471e81a429e1c5fe1947, 91b529dd056e4b7aa9933aa1c68d339f
I20260923 19:30:04.799332 7870 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 19:30:04.799419 7871 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 19:30:05.260442 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:05.259637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.2:34595)
0923 19:30:05.260130 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:05.260235 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":268,"threads_started":1}
I20260923 19:30:05.262382 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:05.261496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:34151)
0923 19:30:05.261957 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:05.262085 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
I20260923 19:30:05.264287 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:05.263547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.3:59537)
0923 19:30:05.263990 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:05.264099 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":248,"threads_started":1}
I20260923 19:30:06.262084 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.261300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.2:43511)
0923 19:30:06.261800 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.261894 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":260,"threads_started":1}
I20260923 19:30:06.264036 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.263202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:59565)
0923 19:30:06.263723 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.263825 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260923 19:30:06.265849 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.265111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.3:40059)
0923 19:30:06.265534 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.265633 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":232,"threads_started":1}
I20260923 19:30:06.805735 7886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:06.805989 7886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } }
I20260923 19:30:06.806731 7886 leader_election.cc:288] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425), 91b529dd056e4b7aa9933aa1c68d339f (127.0.97.61:35879)
I20260923 19:30:06.807375 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d25d7bc72b14cac94ddb530ede55d42" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91b529dd056e4b7aa9933aa1c68d339f" is_pre_election: true
I20260923 19:30:06.807637 7150 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d25d7bc72b14cac94ddb530ede55d42 in term 2.
I20260923 19:30:06.807864 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.807029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54392)
0923 19:30:06.807520 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.807636 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":276,"threads_started":1}
I20260923 19:30:06.808081 7314 leader_election.cc:302] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d25d7bc72b14cac94ddb530ede55d42, 91b529dd056e4b7aa9933aa1c68d339f; no voters:
I20260923 19:30:06.808391 7886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 19:30:06.808553 7886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:30:06.808665 7886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 2 FOLLOWER]: Advancing to term 3
I20260923 19:30:06.809360 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.809007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54398)
0923 19:30:06.809148 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.809226 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 19:30:06.809589 7315 leader_election.cc:334] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
I20260923 19:30:06.811187 7886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } }
I20260923 19:30:06.811713 7886 leader_election.cc:288] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 3 election: Requested vote from peers 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425), 91b529dd056e4b7aa9933aa1c68d339f (127.0.97.61:35879)
I20260923 19:30:06.812129 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.811832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54400)
0923 19:30:06.811949 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.812012 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 19:30:06.812166 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d25d7bc72b14cac94ddb530ede55d42" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91b529dd056e4b7aa9933aa1c68d339f"
I20260923 19:30:06.812410 7150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 2 FOLLOWER]: Advancing to term 3
I20260923 19:30:06.812961 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.812669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54412)
0923 19:30:06.812804 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.812865 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 19:30:06.813120 7315 leader_election.cc:334] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
I20260923 19:30:06.814741 7150 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d25d7bc72b14cac94ddb530ede55d42 in term 3.
I20260923 19:30:06.815128 7314 leader_election.cc:302] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d25d7bc72b14cac94ddb530ede55d42, 91b529dd056e4b7aa9933aa1c68d339f; no voters: 164e149b0070471e81a429e1c5fe1947
I20260923 19:30:06.815392 7886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 19:30:06.815781 7886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [term 3 LEADER]: Becoming Leader. State: Replica: 0d25d7bc72b14cac94ddb530ede55d42, State: Running, Role: LEADER
I20260923 19:30:06.816213 7886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } }
I20260923 19:30:06.818308 7889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d25d7bc72b14cac94ddb530ede55d42. Latest consensus state: current_term: 3 leader_uuid: "0d25d7bc72b14cac94ddb530ede55d42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:06.818642 7889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:06.819603 7891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:30:06.823527 7891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:30:06.824402 7891 catalog_manager.cc:1259] Loaded cluster ID: 7cf1765f85b34a0d83e9aacd359ae234
I20260923 19:30:06.824482 7891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:30:06.825702 7891 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:30:06.826545 7891 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Loaded TSK: 0
I20260923 19:30:06.827484 7891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:30:06.899144 7329 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:30:06.907075 7329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:30:06.912140 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.911661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54450)
0923 19:30:06.911875 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.911967 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 19:30:06.912179 7150 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [term 3 FOLLOWER]: Refusing update from remote peer 0d25d7bc72b14cac94ddb530ede55d42: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 19:30:06.912886 7889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 19:30:06.913175 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.912846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54466)
0923 19:30:06.912994 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.913065 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 19:30:06.913424 7315 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 -> Peer 164e149b0070471e81a429e1c5fe1947 (127.0.97.62:37425): Couldn't send request to peer 164e149b0070471e81a429e1c5fe1947. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 19:30:06.915741 7893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d25d7bc72b14cac94ddb530ede55d42. Latest consensus state: current_term: 3 leader_uuid: "0d25d7bc72b14cac94ddb530ede55d42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:06.915979 7893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:30:06.917639 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0d25d7bc72b14cac94ddb530ede55d42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:06.917984 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:06.918453 7894 mvcc.cc:205] Tried to move back new op lower bound from 7332625641104592896 to 7332625640722649088. Current Snapshot: MvccSnapshot[applied={T|T < 7332625641104592896}]
I20260923 19:30:06.919082 7893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0d25d7bc72b14cac94ddb530ede55d42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "164e149b0070471e81a429e1c5fe1947" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } peers { permanent_uuid: "91b529dd056e4b7aa9933aa1c68d339f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35879 } attrs { promote: false } } peers { permanent_uuid: "0d25d7bc72b14cac94ddb530ede55d42" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44187 } attrs { promote: false } } }
I20260923 19:30:06.919338 7893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b529dd056e4b7aa9933aa1c68d339f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:30:06.935122 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.926744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37351 (local address 127.0.0.1:54056)
0923 19:30:06.926964 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.926986 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:06.927112 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:06.928218 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:06.928223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:06.928252 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:06.928591 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:06.928603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.930265 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.930272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:06.932586 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:06.932604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.934226 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.934232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:06.934467 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:06.934506 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 19:30:06.934578 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 19:30:06.934759 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 19:30:06.936884 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.926920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.0.97.3:37351)
0923 19:30:06.927877 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:06.927883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:06.927902 (+ 19us) server_negotiation.cc:407] Connection header received
0923 19:30:06.927983 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:06.927988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:06.928067 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:06.928749 (+ 682us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:06.928945 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.930114 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.932751 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.934008 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.936178 (+ 2170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:06.936307 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:06.936371 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 19:30:06.936541 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":711,"threads_started":1}
I20260923 19:30:06.938004 7730 tablet_service.cc:1467] Processing CreateTablet for tablet bf26136c9e7d4db2bb4985dd18cc121e (DEFAULT_TABLE table=test-table [id=9b31ae06bc044a76b21330b05d3aaae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:06.938680 7730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf26136c9e7d4db2bb4985dd18cc121e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:06.941772 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.927502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43027 (local address 127.0.0.1:57772)
0923 19:30:06.935480 (+ 7978us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.935497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:06.935624 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:06.935982 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:06.935987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:06.936011 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:06.936199 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:06.936209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.937513 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.937518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:06.939425 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:06.939439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.940944 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.940950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:06.941134 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:06.941172 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 19:30:06.941258 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 19:30:06.941398 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7885,"mutex_wait_us":232,"spinlock_wait_cycles":4352,"thread_start_us":318,"threads_started":1}
I20260923 19:30:06.942668 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.928115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.0.97.1:43027)
0923 19:30:06.934715 (+ 6600us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:06.934722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:06.935717 (+ 995us) server_negotiation.cc:407] Connection header received
0923 19:30:06.935798 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:06.935802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:06.935865 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:06.935948 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:06.936363 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.937374 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.939599 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.940773 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.942003 (+ 1230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:06.942097 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:06.942148 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 19:30:06.942347 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6445,"thread_start_us":6308,"threads_started":1}
I20260923 19:30:06.943250 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.927498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37959 (local address 127.0.0.1:35168)
0923 19:30:06.928258 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.928277 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:06.928409 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:06.935570 (+ 7161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:06.935576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:06.935629 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:06.936040 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:06.936054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.937774 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.937779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:06.940168 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:06.940201 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.942372 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.942377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:06.942546 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:06.942575 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 19:30:06.942632 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 19:30:06.942758 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"spinlock_wait_cycles":2304,"thread_start_us":14546,"threads_started":1}
I20260923 19:30:06.943581 7444 tablet_service.cc:1467] Processing CreateTablet for tablet bf26136c9e7d4db2bb4985dd18cc121e (DEFAULT_TABLE table=test-table [id=9b31ae06bc044a76b21330b05d3aaae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:06.943928 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.927800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.0.97.2:37959)
0923 19:30:06.935107 (+ 7307us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:06.935114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:06.935135 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:30:06.935222 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:06.935228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:06.935337 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:06.936253 (+ 916us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:06.936429 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.937569 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.940418 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:06.941791 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:06.943454 (+ 1663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:06.943562 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:06.943602 (+ 40us) server_negotiation.cc:299] Negotiation successful
0923 19:30:06.943713 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7172,"thread_start_us":7063,"threads_started":1}
I20260923 19:30:06.944003 7444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf26136c9e7d4db2bb4985dd18cc121e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:06.944638 7589 tablet_service.cc:1467] Processing CreateTablet for tablet bf26136c9e7d4db2bb4985dd18cc121e (DEFAULT_TABLE table=test-table [id=9b31ae06bc044a76b21330b05d3aaae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:06.945066 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf26136c9e7d4db2bb4985dd18cc121e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:06.952419 7905 tablet_bootstrap.cc:492] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: Bootstrap starting.
I20260923 19:30:06.953480 7906 tablet_bootstrap.cc:492] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: Bootstrap starting.
I20260923 19:30:06.954775 7905 tablet_bootstrap.cc:654] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:06.955243 7906 tablet_bootstrap.cc:654] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:06.956404 7904 tablet_bootstrap.cc:492] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: Bootstrap starting.
I20260923 19:30:06.958770 7904 tablet_bootstrap.cc:654] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:06.961238 7906 tablet_bootstrap.cc:492] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: No bootstrap required, opened a new log
I20260923 19:30:06.961252 7904 tablet_bootstrap.cc:492] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: No bootstrap required, opened a new log
I20260923 19:30:06.961391 7906 ts_tablet_manager.cc:1397] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 19:30:06.961391 7904 ts_tablet_manager.cc:1397] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 19:30:06.961472 7905 tablet_bootstrap.cc:492] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: No bootstrap required, opened a new log
I20260923 19:30:06.961652 7905 ts_tablet_manager.cc:1397] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 19:30:06.962215 7906 raft_consensus.cc:348] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:06.962260 7904 raft_consensus.cc:348] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:06.962541 7904 raft_consensus.cc:374] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:06.962558 7906 raft_consensus.cc:374] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:06.962653 7906 raft_consensus.cc:729] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaad02bea99c40c8a91cbd47e320579a, State: Initialized, Role: FOLLOWER
I20260923 19:30:06.962667 7904 raft_consensus.cc:729] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130dae76d5ba48beb6873ed7e28d8d13, State: Initialized, Role: FOLLOWER
I20260923 19:30:06.962522 7905 raft_consensus.cc:348] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:06.962767 7905 raft_consensus.cc:374] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:06.962883 7905 raft_consensus.cc:729] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066538205b95484ca562199aa7740b8a, State: Initialized, Role: FOLLOWER
I20260923 19:30:06.962913 7904 consensus_queue.cc:260] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:06.962954 7906 consensus_queue.cc:260] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:06.963157 7905 consensus_queue.cc:260] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:06.964325 7905 ts_tablet_manager.cc:1428] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 19:30:06.967535 7906 ts_tablet_manager.cc:1428] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 19:30:06.971582 7904 ts_tablet_manager.cc:1428] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260923 19:30:06.982191 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.967693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.2:36339)
0923 19:30:06.981547 (+ 13854us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.981764 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13635,"thread_start_us":14093,"threads_started":1}
I20260923 19:30:06.983168 7795 heartbeater.cc:503] Master 127.0.97.60:44187 was elected leader, sending a full tablet report...
I20260923 19:30:06.983872 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.972287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.3:58193)
0923 19:30:06.983024 (+ 10737us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.983188 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10569,"thread_start_us":10399,"threads_started":1}
I20260923 19:30:06.984151 7654 heartbeater.cc:503] Master 127.0.97.60:44187 was elected leader, sending a full tablet report...
I20260923 19:30:06.985107 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.965036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:52249)
0923 19:30:06.984655 (+ 19619us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.984823 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19480,"thread_start_us":19307,"threads_started":1}
I20260923 19:30:06.985538 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.985058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.3:60503)
0923 19:30:06.985274 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.985361 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260923 19:30:06.986392 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.985731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.2:58407)
0923 19:30:06.986110 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.986210 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
I20260923 19:30:06.987257 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:06.986814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:43543)
0923 19:30:06.987051 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:06.987126 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260923 19:30:06.987319 7509 heartbeater.cc:503] Master 127.0.97.60:44187 was elected leader, sending a full tablet report...
I20260923 19:30:07.116889 7912 raft_consensus.cc:482] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:07.117148 7912 raft_consensus.cc:504] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:07.117877 7912 leader_election.cc:288] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351), aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959)
I20260923 19:30:07.118721 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf26136c9e7d4db2bb4985dd18cc121e" candidate_uuid: "066538205b95484ca562199aa7740b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaad02bea99c40c8a91cbd47e320579a" is_pre_election: true
I20260923 19:30:07.118773 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf26136c9e7d4db2bb4985dd18cc121e" candidate_uuid: "066538205b95484ca562199aa7740b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130dae76d5ba48beb6873ed7e28d8d13" is_pre_election: true
I20260923 19:30:07.119027 7609 raft_consensus.cc:2454] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 066538205b95484ca562199aa7740b8a in term 0.
I20260923 19:30:07.119092 7750 raft_consensus.cc:2454] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 066538205b95484ca562199aa7740b8a in term 0.
I20260923 19:30:07.119547 7399 leader_election.cc:302] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066538205b95484ca562199aa7740b8a, aaad02bea99c40c8a91cbd47e320579a; no voters:
I20260923 19:30:07.119884 7912 raft_consensus.cc:2781] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:30:07.120007 7912 raft_consensus.cc:482] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:30:07.120100 7912 raft_consensus.cc:3037] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:07.122430 7912 raft_consensus.cc:504] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:07.123250 7912 leader_election.cc:288] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [CANDIDATE]: Term 1 election: Requested vote from peers 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351), aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959)
I20260923 19:30:07.123980 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf26136c9e7d4db2bb4985dd18cc121e" candidate_uuid: "066538205b95484ca562199aa7740b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaad02bea99c40c8a91cbd47e320579a"
I20260923 19:30:07.124004 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf26136c9e7d4db2bb4985dd18cc121e" candidate_uuid: "066538205b95484ca562199aa7740b8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130dae76d5ba48beb6873ed7e28d8d13"
I20260923 19:30:07.124209 7609 raft_consensus.cc:3037] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:07.124221 7750 raft_consensus.cc:3037] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:07.127274 7609 raft_consensus.cc:2454] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 066538205b95484ca562199aa7740b8a in term 1.
I20260923 19:30:07.127275 7750 raft_consensus.cc:2454] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 066538205b95484ca562199aa7740b8a in term 1.
I20260923 19:30:07.127892 7399 leader_election.cc:302] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 066538205b95484ca562199aa7740b8a, 130dae76d5ba48beb6873ed7e28d8d13, aaad02bea99c40c8a91cbd47e320579a; no voters:
I20260923 19:30:07.128319 7912 raft_consensus.cc:2781] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:07.128871 7912 raft_consensus.cc:686] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 1 LEADER]: Becoming Leader. State: Replica: 066538205b95484ca562199aa7740b8a, State: Running, Role: LEADER
I20260923 19:30:07.129567 7912 consensus_queue.cc:237] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } }
I20260923 19:30:07.130030 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.129506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:47775)
0923 19:30:07.129717 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:07.129848 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 19:30:07.133117 7330 catalog_manager.cc:5495] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 066538205b95484ca562199aa7740b8a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "066538205b95484ca562199aa7740b8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "066538205b95484ca562199aa7740b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20260923 19:30:07.192610 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.187440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.0.97.1:43027)
0923 19:30:07.187662 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:07.187668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:07.187687 (+ 19us) server_negotiation.cc:407] Connection header received
0923 19:30:07.187886 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:07.187891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:07.187980 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:07.188073 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:07.188545 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:07.189388 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:07.190872 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:07.191324 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:07.191460 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:07.191845 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:07.191968 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:07.192039 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:07.192222 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 19:30:07.192300 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 19:30:07.196154 7609 raft_consensus.cc:1261] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [term 1 FOLLOWER]: Refusing update from remote peer 066538205b95484ca562199aa7740b8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:07.196153 7750 raft_consensus.cc:1261] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 066538205b95484ca562199aa7740b8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:07.196906 7912 consensus_queue.cc:1035] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaad02bea99c40c8a91cbd47e320579a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:07.197184 7916 consensus_queue.cc:1035] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "130dae76d5ba48beb6873ed7e28d8d13" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:07.200356 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.199746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.3:50189)
0923 19:30:07.200008 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:07.200137 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260923 19:30:07.202967 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.202388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.2:34479)
0923 19:30:07.202654 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:07.202777 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 19:30:07.208388 7921 mvcc.cc:205] Tried to move back new op lower bound from 7332625642268835840 to 7332625642005995520. Current Snapshot: MvccSnapshot[applied={T|T < 7332625642268835840}]
I20260923 19:30:07.209830 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.208888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.2:34175)
0923 19:30:07.209482 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:07.209601 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453}
I20260923 19:30:07.211429 7919 mvcc.cc:205] Tried to move back new op lower bound from 7332625642268835840 to 7332625642005995520. Current Snapshot: MvccSnapshot[applied={T|T < 7332625642268835840}]
I20260923 19:30:07.218587 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.217976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:51513)
0923 19:30:07.218233 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:07.218362 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 19:30:07.219504 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.218666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.3:46055)
0923 19:30:07.219224 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:07.219346 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450}
I20260923 19:30:07.223109 7330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:07.223439 7330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:07.224504 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.221603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.97.1:36237)
0923 19:30:07.224204 (+ 2601us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:07.224290 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2468}
I20260923 19:30:07.230544 7330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d25d7bc72b14cac94ddb530ede55d42: Sending DeleteTablet for 3 replicas of tablet bf26136c9e7d4db2bb4985dd18cc121e
I20260923 19:30:07.231819 7729 tablet_service.cc:1514] Processing DeleteTablet for tablet bf26136c9e7d4db2bb4985dd18cc121e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:07 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:54056
I20260923 19:30:07.231884 7730 tablet_service.cc:1514] Processing DeleteTablet for tablet bf26136c9e7d4db2bb4985dd18cc121e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:07 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:54056
I20260923 19:30:07.231902 7443 tablet_service.cc:1514] Processing DeleteTablet for tablet bf26136c9e7d4db2bb4985dd18cc121e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:07 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57772
I20260923 19:30:07.232211 7444 tablet_service.cc:1514] Processing DeleteTablet for tablet bf26136c9e7d4db2bb4985dd18cc121e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:07 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57772
I20260923 19:30:07.232264 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet bf26136c9e7d4db2bb4985dd18cc121e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:07 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35168
I20260923 19:30:07.235692 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 7317
I20260923 19:30:07.243027 7929 tablet_replica.cc:330] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: stopping tablet replica
I20260923 19:30:07.243245 7927 tablet_replica.cc:330] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: stopping tablet replica
I20260923 19:30:07.243310 7929 raft_consensus.cc:2229] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:07.243464 7927 raft_consensus.cc:2229] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:30:07.244305 7927 raft_consensus.cc:2258] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:07.245081 7929 raft_consensus.cc:2258] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:07.245556 7927 ts_tablet_manager.cc:1905] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:07.246994 7929 ts_tablet_manager.cc:1905] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:07.250846 7928 tablet_replica.cc:330] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: stopping tablet replica
I20260923 19:30:07.251084 7928 raft_consensus.cc:2229] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:07.251410 7928 raft_consensus.cc:2258] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:07.252796 7928 ts_tablet_manager.cc:1905] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:07.252851 7927 ts_tablet_manager.cc:1918] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:07.253091 7927 log.cc:1196] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/bf26136c9e7d4db2bb4985dd18cc121e
I20260923 19:30:07.253693 7927 ts_tablet_manager.cc:1939] T bf26136c9e7d4db2bb4985dd18cc121e P 066538205b95484ca562199aa7740b8a: Deleting consensus metadata
I20260923 19:30:07.255223 7929 ts_tablet_manager.cc:1918] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:07.255268 7316 catalog_manager.cc:4841] TS 066538205b95484ca562199aa7740b8a (127.0.97.1:43027): tablet bf26136c9e7d4db2bb4985dd18cc121e (table test-table [id=9b31ae06bc044a76b21330b05d3aaae8]) successfully deleted
I20260923 19:30:07.255401 7929 log.cc:1196] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/bf26136c9e7d4db2bb4985dd18cc121e
I20260923 19:30:07.255849 7929 ts_tablet_manager.cc:1939] T bf26136c9e7d4db2bb4985dd18cc121e P aaad02bea99c40c8a91cbd47e320579a: Deleting consensus metadata
I20260923 19:30:07.255946 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.237433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.0.97.1:43027)
0923 19:30:07.237619 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:07.237626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:07.241518 (+ 3892us) server_negotiation.cc:407] Connection header received
0923 19:30:07.241765 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:07.241796 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:07.241956 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:07.242086 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:07.244024 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:07.245105 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:07.246378 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:07.246803 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:07.246914 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:07.249271 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:07.249284 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:07.249301 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:07.249335 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:07.252699 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:07.253466 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:07.253475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:07.253479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:07.253665 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:07.254072 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:07.254078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:07.254081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:07.254543 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:07.254841 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:07.255372 (+ 531us) server_negotiation.cc:299] Negotiation successful
0923 19:30:07.255603 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 19:30:07.256158 7316 catalog_manager.cc:4808] TS 066538205b95484ca562199aa7740b8a (127.0.97.1:43027): delete failed for tablet bf26136c9e7d4db2bb4985dd18cc121e because the tablet was not found. No further retry: Not found: Tablet not found: bf26136c9e7d4db2bb4985dd18cc121e
I20260923 19:30:07.257009 7315 catalog_manager.cc:4841] TS aaad02bea99c40c8a91cbd47e320579a (127.0.97.2:37959): tablet bf26136c9e7d4db2bb4985dd18cc121e (table test-table [id=9b31ae06bc044a76b21330b05d3aaae8]) successfully deleted
I20260923 19:30:07.257222 7504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:07.261066 7928 ts_tablet_manager.cc:1918] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:07.261264 7928 log.cc:1196] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/bf26136c9e7d4db2bb4985dd18cc121e
I20260923 19:30:07.261731 7928 ts_tablet_manager.cc:1939] T bf26136c9e7d4db2bb4985dd18cc121e P 130dae76d5ba48beb6873ed7e28d8d13: Deleting consensus metadata
I20260923 19:30:07.262956 7314 catalog_manager.cc:4841] TS 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351): tablet bf26136c9e7d4db2bb4985dd18cc121e (table test-table [id=9b31ae06bc044a76b21330b05d3aaae8]) successfully deleted
W20260923 19:30:07.263399 7314 catalog_manager.cc:4808] TS 130dae76d5ba48beb6873ed7e28d8d13 (127.0.97.3:37351): delete failed for tablet bf26136c9e7d4db2bb4985dd18cc121e because the tablet was not found. No further retry: Not found: Tablet not found: bf26136c9e7d4db2bb4985dd18cc121e
I20260923 19:30:07.389808 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 7317
I20260923 19:30:07.410615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 7526
I20260923 19:30:07.423707 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.411415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.97.2:37959)
0923 19:30:07.411716 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:07.411721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:07.411744 (+ 23us) server_negotiation.cc:407] Connection header received
0923 19:30:07.411891 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:07.411895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:07.411960 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:07.412045 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:07.413306 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:07.414005 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:07.415294 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:07.415682 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:07.415791 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:07.418277 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:07.418334 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:07.418355 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:07.418393 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:07.421114 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:07.421733 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:07.421740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:07.421742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:07.421882 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:07.422213 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:07.422217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:07.422219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:07.422612 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:07.422892 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:07.423168 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 19:30:07.423379 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 19:30:07.424451 7649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:07.470866 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.469901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54478)
0923 19:30:07.470537 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:07.470646 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":381,"threads_started":1}
I20260923 19:30:07.472467 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.472111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:54482)
0923 19:30:07.472285 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:07.472361 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 19:30:07.519670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 7526
I20260923 19:30:07.536151 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 7667
I20260923 19:30:07.550886 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:07.537016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.0.97.3:37351)
0923 19:30:07.537635 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:07.537640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:07.537682 (+ 42us) server_negotiation.cc:407] Connection header received
0923 19:30:07.537737 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:07.537741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:07.537816 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:07.537927 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:07.539816 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:07.540594 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:07.541988 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:07.542511 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:07.542613 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:07.545441 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:07.545456 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:07.545473 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:07.545517 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:07.548286 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:07.549013 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:07.549021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:07.549024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:07.549160 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:07.549474 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:07.549477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:07.549479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:07.549832 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:07.550062 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:07.550329 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 19:30:07.550571 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":316,"threads_started":1}
I20260923 19:30:07.551729 7790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:07.654278 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 7667
I20260923 19:30:07.670759 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 6996
I20260923 19:30:07.671990 7170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:07.802158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 6996
I20260923 19:30:07.819591 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 7096
I20260923 19:30:07.820864 7360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:07.956681 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 7096
2026-09-23T19:30:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12203 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:30:08 dist-test-slave-kxfb krb5kdc[7941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:30:08 dist-test-slave-kxfb krb5kdc[7941](info): set up 2 sockets
Sep 23 19:30:08 dist-test-slave-kxfb krb5kdc[7941](info): commencing operation
krb5kdc: starting...
W20260923 19:30:08.987190 388 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:30:09 dist-test-slave-kxfb krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:30:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260923 19:30:09.233441 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46615
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:09.605093 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:09.605499 7954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:09.605630 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:09.613507 7954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:30:09.613657 7954 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 19:30:09.613718 7954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:09.613807 7954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:09.613878 7954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:30:09.613970 7954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:30:09.624960 7954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46615
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:09.626648 7954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:09.628254 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:09.641825 7960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:09.641908 7962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:09.641939 7959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:09.642300 7954 server_base.cc:1034] running on GCE node
I20260923 19:30:09.643438 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:09.645215 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:09.646410 7954 hybrid_clock.cc:648] HybridClock initialized: now 1790191809646344 us; error 52 us; skew 500 ppm
I20260923 19:30:09.646763 7954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:09 dist-test-slave-kxfb krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:09.650252 7954 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260923 19:30:09.651908 7954 webserver.cc:460] Webserver started at http://127.0.97.62:43193/ using document root <none> and password file <none>
I20260923 19:30:09.652699 7954 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:09.652787 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:09.653098 7954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:09.656533 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "64ff20b6bb7745c0854376598e49013a"
format_stamp: "Formatted at 2026-09-23 19:30:09 on dist-test-slave-kxfb"
server_key: "270a0a45f5cc6330b41d5c0f742b8914"
server_key_iv: "25cc34af5941f9dac3e781bed6a559a6"
server_key_version: "encryptionkey@0"
I20260923 19:30:09.657364 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "64ff20b6bb7745c0854376598e49013a"
format_stamp: "Formatted at 2026-09-23 19:30:09 on dist-test-slave-kxfb"
server_key: "270a0a45f5cc6330b41d5c0f742b8914"
server_key_iv: "25cc34af5941f9dac3e781bed6a559a6"
server_key_version: "encryptionkey@0"
I20260923 19:30:09.663335 7954 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 19:30:09.667135 7969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:09.668694 7954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 19:30:09.668859 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "64ff20b6bb7745c0854376598e49013a"
format_stamp: "Formatted at 2026-09-23 19:30:09 on dist-test-slave-kxfb"
server_key: "270a0a45f5cc6330b41d5c0f742b8914"
server_key_iv: "25cc34af5941f9dac3e781bed6a559a6"
server_key_version: "encryptionkey@0"
I20260923 19:30:09.669095 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:09.689419 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:09.694418 7954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:09.694684 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:09.717152 7954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46615
I20260923 19:30:09.717176 8030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46615 every 8 connection(s)
I20260923 19:30:09.718796 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:30:09.719426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 7954
I20260923 19:30:09.719682 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:30:09.720513 388 external_mini_cluster.cc:1428] Setting key 0d20206fdfe6491a9e3776255e01a33e
I20260923 19:30:09.724324 8031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 19:30:09 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:30:09.736573 8031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a: Bootstrap starting.
I20260923 19:30:09.740744 8031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:09.742084 8031 log.cc:824] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:09.744328 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:09.722938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.0.97.62:46615)
0923 19:30:09.723771 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:09.723785 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:09.723831 (+ 46us) server_negotiation.cc:407] Connection header received
0923 19:30:09.725692 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:09.725718 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:09.726161 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:09.726622 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:09.728210 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:09.729715 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:09.731078 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:09.731823 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:09.731996 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:09.736280 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:09.736289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:09.736309 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:09.736355 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:09.739533 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:09.740422 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:09.740430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:09.740437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:09.740625 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:09.741067 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:09.741071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:09.741073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:09.741696 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:09.741951 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:09.742474 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 19:30:09.742990 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":370,"threads_started":1}
I20260923 19:30:09.745620 8031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a: No bootstrap required, opened a new log
I20260923 19:30:09.753963 8031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ff20b6bb7745c0854376598e49013a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46615 } }
I20260923 19:30:09.754504 8031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:09.754606 8031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ff20b6bb7745c0854376598e49013a, State: Initialized, Role: FOLLOWER
I20260923 19:30:09.755245 8031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ff20b6bb7745c0854376598e49013a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46615 } }
I20260923 19:30:09.755481 8031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:30:09.755611 8031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:30:09.755782 8031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:09.758196 8031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ff20b6bb7745c0854376598e49013a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46615 } }
I20260923 19:30:09.758864 8031 leader_election.cc:302] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64ff20b6bb7745c0854376598e49013a; no voters:
I20260923 19:30:09.759518 8031 leader_election.cc:288] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:30:09.759629 8036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:09.760316 8036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [term 1 LEADER]: Becoming Leader. State: Replica: 64ff20b6bb7745c0854376598e49013a, State: Running, Role: LEADER
I20260923 19:30:09.760838 8036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ff20b6bb7745c0854376598e49013a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46615 } }
I20260923 19:30:09.761752 8031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:30:09.763900 8038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64ff20b6bb7745c0854376598e49013a. Latest consensus state: current_term: 1 leader_uuid: "64ff20b6bb7745c0854376598e49013a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ff20b6bb7745c0854376598e49013a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46615 } } }
I20260923 19:30:09.764078 8037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64ff20b6bb7745c0854376598e49013a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ff20b6bb7745c0854376598e49013a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46615 } } }
I20260923 19:30:09.764338 8038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:09.764389 8037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:09.765913 8041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:30:09.771538 8041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:30:09.780234 8053 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 19:30:09.781658 8041 catalog_manager.cc:1347] Generated new cluster ID: 24447d29cab14ea58372071355297e8b
I20260923 19:30:09.781785 8041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:30:09.798182 8041 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:30:09.800407 8041 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:30:09.827582 8041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a: Generated new TSK 0
I20260923 19:30:09.829027 8041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260923 19:30:09.903203 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46615
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:10.265731 8057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:10.266038 8057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:10.266175 8057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:10.273602 8057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:10.273730 8057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:10.273846 8057 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 19:30:10.273960 8057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:30:10.283913 8057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:10.285279 8057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:10.286638 8057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:10.300770 8063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:10.300732 8062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:10.300825 8065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:10.301687 8057 server_base.cc:1034] running on GCE node
I20260923 19:30:10.302515 8057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:10.303723 8057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:10.304975 8057 hybrid_clock.cc:648] HybridClock initialized: now 1790191810304898 us; error 46 us; skew 500 ppm
I20260923 19:30:10.305315 8057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:10 dist-test-slave-kxfb krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191810, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:10.308802 8057 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 19:30:10.310525 8057 webserver.cc:460] Webserver started at http://127.0.97.1:41503/ using document root <none> and password file <none>
I20260923 19:30:10.311343 8057 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:10.311456 8057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:10.311800 8057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:10.315182 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "6aabbf67c1df49269bea3b750e81ebe8"
format_stamp: "Formatted at 2026-09-23 19:30:10 on dist-test-slave-kxfb"
server_key: "e69646dc20a33843ae057304c9004c28"
server_key_iv: "23618fee03562ae991759333fb168ec7"
server_key_version: "encryptionkey@0"
I20260923 19:30:10.316056 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6aabbf67c1df49269bea3b750e81ebe8"
format_stamp: "Formatted at 2026-09-23 19:30:10 on dist-test-slave-kxfb"
server_key: "e69646dc20a33843ae057304c9004c28"
server_key_iv: "23618fee03562ae991759333fb168ec7"
server_key_version: "encryptionkey@0"
I20260923 19:30:10.322078 8057 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260923 19:30:10.326272 8072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:10.328050 8057 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:30:10.328239 8057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "6aabbf67c1df49269bea3b750e81ebe8"
format_stamp: "Formatted at 2026-09-23 19:30:10 on dist-test-slave-kxfb"
server_key: "e69646dc20a33843ae057304c9004c28"
server_key_iv: "23618fee03562ae991759333fb168ec7"
server_key_version: "encryptionkey@0"
I20260923 19:30:10.328492 8057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:10.351724 8057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:10.356833 8057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:10.357148 8057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:10.360484 8057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:10.360607 8057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:10.360750 8057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:10.360878 8057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 23 19:30:10 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191810, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:30:10.387534 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:10.361072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40461 (local address 127.0.97.62:46615)
0923 19:30:10.364441 (+ 3369us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:10.364450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:10.364493 (+ 43us) server_negotiation.cc:407] Connection header received
0923 19:30:10.366091 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:10.366101 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:10.366227 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:10.366424 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:10.368882 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.369839 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.373855 (+ 4016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.374438 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.374611 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:10.380052 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:10.380057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:10.380061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:10.380097 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:10.383712 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:10.384554 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:10.384561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:10.384566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:10.384770 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:10.385228 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:10.385234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:10.385237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:10.385665 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:10.385821 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:10.386781 (+ 960us) server_negotiation.cc:299] Negotiation successful
0923 19:30:10.387075 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3198,"thread_start_us":3043,"threads_started":1}
I20260923 19:30:10.388656 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:10.362028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46615 (local address 127.0.97.1:40461)
0923 19:30:10.363442 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:10.363514 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:10.365760 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:10.366704 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:10.366717 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:10.367468 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:10.368608 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:10.368647 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.370110 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.370117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:10.373491 (+ 3374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:10.373523 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.374817 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.374826 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:10.375203 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:10.376328 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:10.376384 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:10.379776 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:10.383932 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:10.383942 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:10.383957 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:10.384387 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:10.384899 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:10.384902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:10.384904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:10.385114 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:10.385809 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:10.385815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:10.386244 (+ 429us) client_negotiation.cc:769] Sending connection context
0923 19:30:10.386631 (+ 387us) client_negotiation.cc:240] Negotiation successful
0923 19:30:10.387269 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":543,"threads_started":1}
I20260923 19:30:10.415536 8057 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35179
I20260923 19:30:10.415792 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:10.396206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53819 (local address 127.0.97.62:46615)
0923 19:30:10.396451 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:10.396459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:10.398920 (+ 2461us) server_negotiation.cc:407] Connection header received
0923 19:30:10.399075 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:10.399082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:10.399201 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:10.399350 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:10.401389 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.402693 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.404945 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.405411 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.405530 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:10.408823 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:10.408834 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:10.408840 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:10.408920 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:10.412158 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:10.413193 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:10.413198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:10.413200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:10.413343 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:10.413808 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:10.413814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:10.413817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:10.414237 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:10.414439 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:10.415052 (+ 613us) server_negotiation.cc:299] Negotiation successful
0923 19:30:10.415313 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 19:30:10.415548 8192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35179 every 8 connection(s)
I20260923 19:30:10.416114 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:10.396153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46615 (local address 127.0.97.1:53819)
0923 19:30:10.398794 (+ 2641us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:10.398814 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:10.398945 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:10.399596 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:10.399602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:10.400446 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:10.401141 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:10.401160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.402936 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.402941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:10.404731 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:10.404750 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.405607 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.405614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:10.405889 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:10.406869 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:10.406900 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:10.408430 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:10.412341 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:10.412349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:10.412354 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:10.413055 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:10.413463 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:10.413468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:10.413471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:10.413629 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:10.414402 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:10.414408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:10.414696 (+ 288us) client_negotiation.cc:769] Sending connection context
0923 19:30:10.414909 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 19:30:10.415350 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2507,"thread_start_us":2377,"threads_started":1}
I20260923 19:30:10.417320 8057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:30:10.419658 8193 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46615
I20260923 19:30:10.420089 8193 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:10.420867 8193 heartbeater.cc:511] Master 127.0.97.62:46615 requested a full tablet report, sending...
I20260923 19:30:10.422243 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 8057
I20260923 19:30:10.422474 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:30:10.423097 388 external_mini_cluster.cc:1428] Setting key ccbc6cf60a891269842f592ee32a6602
I20260923 19:30:10.423352 7986 ts_manager.cc:194] Registered new tserver with Master: 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179)
I20260923 19:30:10.425608 7986 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40461
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260923 19:30:10.474403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46615
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:10.854048 8197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:10.854446 8197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:10.854573 8197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:10.862331 8197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:10.862465 8197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:10.862568 8197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 19:30:10.862658 8197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:30:10.873306 8197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:10.874895 8197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:10.876384 8197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:10.890842 8202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:10.890949 8205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:10.890870 8203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:10.891932 8197 server_base.cc:1034] running on GCE node
I20260923 19:30:10.892635 8197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:10.893715 8197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:10.894979 8197 hybrid_clock.cc:648] HybridClock initialized: now 1790191810894888 us; error 58 us; skew 500 ppm
I20260923 19:30:10.895337 8197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:10 dist-test-slave-kxfb krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191810, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:10.898973 8197 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 19:30:10.900617 8197 webserver.cc:460] Webserver started at http://127.0.97.2:45431/ using document root <none> and password file <none>
I20260923 19:30:10.901443 8197 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:10.901556 8197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:10.901904 8197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:10.905277 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31"
format_stamp: "Formatted at 2026-09-23 19:30:10 on dist-test-slave-kxfb"
server_key: "e2458bf4d7ce87ab149c861c180315d1"
server_key_iv: "d977763a2726f1c87d22b08df7c3c09a"
server_key_version: "encryptionkey@0"
I20260923 19:30:10.906142 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31"
format_stamp: "Formatted at 2026-09-23 19:30:10 on dist-test-slave-kxfb"
server_key: "e2458bf4d7ce87ab149c861c180315d1"
server_key_iv: "d977763a2726f1c87d22b08df7c3c09a"
server_key_version: "encryptionkey@0"
I20260923 19:30:10.912196 8197 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 19:30:10.916415 8212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:10.918174 8197 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 19:30:10.918383 8197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31"
format_stamp: "Formatted at 2026-09-23 19:30:10 on dist-test-slave-kxfb"
server_key: "e2458bf4d7ce87ab149c861c180315d1"
server_key_iv: "d977763a2726f1c87d22b08df7c3c09a"
server_key_version: "encryptionkey@0"
I20260923 19:30:10.918622 8197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:10.930832 8197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:10.935273 8197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:10.935534 8197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:10.938498 8197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:10.938591 8197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:10.938702 8197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:10.938804 8197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 19:30:10 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191810, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:30:10.969805 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:10.939558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35993 (local address 127.0.97.62:46615)
0923 19:30:10.940085 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:10.940090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:10.941368 (+ 1278us) server_negotiation.cc:407] Connection header received
0923 19:30:10.944210 (+ 2842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:10.944216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:10.944319 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:10.944444 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:10.947658 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.948766 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.951719 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.952522 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.952681 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:10.959151 (+ 6470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:10.959160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:10.959167 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:10.959224 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:10.963239 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:10.964711 (+ 1472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:10.964719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:10.964723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:10.964886 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:10.965496 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:10.965502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:10.965506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:10.965945 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:10.966112 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:10.969024 (+ 2912us) server_negotiation.cc:299] Negotiation successful
0923 19:30:10.969322 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":298,"threads_started":1}
I20260923 19:30:10.970170 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:10.940260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46615 (local address 127.0.97.2:35993)
0923 19:30:10.941173 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:10.941227 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:10.943744 (+ 2517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:10.944901 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:10.944917 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:10.946084 (+ 1167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:10.947286 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:10.947325 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.948991 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.948998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:10.951510 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:10.951530 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.953110 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.953118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:10.953526 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:10.954633 (+ 1107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:10.954685 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:10.958785 (+ 4100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:10.963532 (+ 4747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:10.963549 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:10.963575 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:10.964527 (+ 952us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:10.965015 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:10.965020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:10.965025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:10.965360 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:10.966090 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:10.966105 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:10.966813 (+ 708us) client_negotiation.cc:769] Sending connection context
0923 19:30:10.967252 (+ 439us) client_negotiation.cc:240] Negotiation successful
0923 19:30:10.967998 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":359,"threads_started":1}
I20260923 19:30:10.981963 8197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41091
I20260923 19:30:10.982029 8332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41091 every 8 connection(s)
I20260923 19:30:10.983419 8197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:30:10.986032 8333 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46615
I20260923 19:30:10.986382 8333 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:10.987408 8333 heartbeater.cc:511] Master 127.0.97.62:46615 requested a full tablet report, sending...
I20260923 19:30:10.988488 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 8197
I20260923 19:30:10.988679 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:30:10.989163 388 external_mini_cluster.cc:1428] Setting key c86fa1defde4ad813eb6ac3632293ffb
I20260923 19:30:10.989485 7986 ts_manager.cc:194] Registered new tserver with Master: 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091)
I20260923 19:30:10.990618 7986 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35993
I20260923 19:30:10.995160 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:10.979911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37969 (local address 127.0.97.62:46615)
0923 19:30:10.980428 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:10.980436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:10.980851 (+ 415us) server_negotiation.cc:407] Connection header received
0923 19:30:10.981176 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:10.981184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:10.981293 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:10.981379 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:10.982959 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.983743 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.985717 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.986408 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.986547 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:10.988952 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:10.988959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:10.988965 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:10.989006 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:10.992453 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:10.993204 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:10.993209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:10.993214 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:10.993329 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:10.993670 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:10.993675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:10.993679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:10.994012 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:10.994156 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:10.994587 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 19:30:10.994815 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339}
I20260923 19:30:10.995247 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:10.979983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46615 (local address 127.0.97.2:37969)
0923 19:30:10.980699 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:10.980730 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:10.980981 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:10.981632 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:10.981637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:10.982410 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:10.982799 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:10.982810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.983944 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.983950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:10.985551 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:10.985569 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:10.986648 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:10.986656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:10.986875 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:10.987574 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:10.987598 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:10.988727 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:10.992616 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:10.992630 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:10.992633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:10.993088 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:10.993429 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:10.993434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:10.993438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:10.993563 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:10.994128 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:10.994133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:10.994352 (+ 219us) client_negotiation.cc:769] Sending connection context
0923 19:30:10.994485 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 19:30:10.994833 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":461,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260923 19:30:11.043036 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46615
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:11.385244 8337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:11.385550 8337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:11.385672 8337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:11.393138 8337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:11.393262 8337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:11.393363 8337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 19:30:11.393476 8337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:30:11.403527 8337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:11.404831 8337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:11.406184 8337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:11.420107 8343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:11.420156 8342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:11.420571 8345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:11.421082 8337 server_base.cc:1034] running on GCE node
I20260923 19:30:11.421737 8337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:11.422755 8337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:11.424021 8337 hybrid_clock.cc:648] HybridClock initialized: now 1790191811423955 us; error 40 us; skew 500 ppm
I20260923 19:30:11.424340 8337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:11 dist-test-slave-kxfb krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191811, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:11.427874 8337 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 19:30:11.429488 8337 webserver.cc:460] Webserver started at http://127.0.97.3:43185/ using document root <none> and password file <none>
I20260923 19:30:11.429507 8193 heartbeater.cc:503] Master 127.0.97.62:46615 was elected leader, sending a full tablet report...
I20260923 19:30:11.430263 8337 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:11.430399 8337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:11.430763 8337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:11.433910 8337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "c4d4138521c24c8e8b4a9772aa0b6a02"
format_stamp: "Formatted at 2026-09-23 19:30:11 on dist-test-slave-kxfb"
server_key: "fb0a07c5d16b0dc15aeb2d92dacdcad1"
server_key_iv: "0a8ccb5245ff0b44b62df512e4aa070f"
server_key_version: "encryptionkey@0"
I20260923 19:30:11.434721 8337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c4d4138521c24c8e8b4a9772aa0b6a02"
format_stamp: "Formatted at 2026-09-23 19:30:11 on dist-test-slave-kxfb"
server_key: "fb0a07c5d16b0dc15aeb2d92dacdcad1"
server_key_iv: "0a8ccb5245ff0b44b62df512e4aa070f"
server_key_version: "encryptionkey@0"
I20260923 19:30:11.440227 8337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 19:30:11.444186 8352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:11.445883 8337 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:30:11.446074 8337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "c4d4138521c24c8e8b4a9772aa0b6a02"
format_stamp: "Formatted at 2026-09-23 19:30:11 on dist-test-slave-kxfb"
server_key: "fb0a07c5d16b0dc15aeb2d92dacdcad1"
server_key_iv: "0a8ccb5245ff0b44b62df512e4aa070f"
server_key_version: "encryptionkey@0"
I20260923 19:30:11.446312 8337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:11.473862 8337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:11.478220 8337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:11.478499 8337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:11.481362 8337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:11.481448 8337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:11.481585 8337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:11.481660 8337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 19:30:11 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191811, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:30:11.506712 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.482452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51539 (local address 127.0.97.62:46615)
0923 19:30:11.482628 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.482632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.484425 (+ 1793us) server_negotiation.cc:407] Connection header received
0923 19:30:11.487440 (+ 3015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.487445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.487525 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.487591 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:11.490697 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.491672 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.493484 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.493912 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.494023 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.499598 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:11.499604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:11.499608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:11.499648 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:11.502464 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:11.503624 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:11.503633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:11.503637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:11.503897 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:11.504417 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:11.504423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:11.504426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:11.504888 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:11.505070 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.505962 (+ 892us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.506316 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 19:30:11.507993 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.483140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46615 (local address 127.0.97.3:51539)
0923 19:30:11.484194 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:11.484268 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:11.486996 (+ 2728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:11.488243 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:11.488261 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:11.489357 (+ 1096us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:11.490484 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.490521 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.491859 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.491864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.493341 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.493354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.494167 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.494174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.494454 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.495361 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:11.495405 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:11.499240 (+ 3835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:11.502640 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:11.502652 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:11.502678 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:11.503406 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:11.504043 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:11.504047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:11.504049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:11.504262 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:11.505051 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:11.505058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:11.505505 (+ 447us) client_negotiation.cc:769] Sending connection context
0923 19:30:11.505775 (+ 270us) client_negotiation.cc:240] Negotiation successful
0923 19:30:11.506390 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":370,"threads_started":1}
I20260923 19:30:11.530200 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.514391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47193 (local address 127.0.97.62:46615)
0923 19:30:11.514635 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.514644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.515187 (+ 543us) server_negotiation.cc:407] Connection header received
0923 19:30:11.515426 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.515430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.515502 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.515604 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:11.517219 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.518470 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.520582 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.521299 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.521454 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.523932 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:11.523939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:11.523945 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:11.523989 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:11.527290 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:11.528209 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:11.528215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:11.528218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:11.528362 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:11.528693 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:11.528699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:11.528702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:11.529011 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:11.529144 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.529569 (+ 425us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.529787 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 19:30:11.530258 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.514224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46615 (local address 127.0.97.3:47193)
0923 19:30:11.515042 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:11.515070 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:11.515287 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:11.515668 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:11.515673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:11.516421 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:11.517015 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.517032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.518693 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.518699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.520413 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.520430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.521550 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.521555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.521711 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.522405 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:11.522423 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:11.523642 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:11.527469 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:11.527473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:11.527476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:11.528095 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:11.528464 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:11.528470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:11.528474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:11.528597 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:11.529120 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:11.529125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:11.529312 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 19:30:11.529444 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 19:30:11.529801 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":413,"threads_started":1}
I20260923 19:30:11.543833 8337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43099
I20260923 19:30:11.543867 8471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43099 every 8 connection(s)
I20260923 19:30:11.545121 8337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:30:11.547199 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 8337
I20260923 19:30:11.547384 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:30:11.547683 8472 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46615
I20260923 19:30:11.547835 388 external_mini_cluster.cc:1428] Setting key d1202deffb4127eb70c107b8f0e7e0fb
I20260923 19:30:11.547973 8472 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:11.548718 8472 heartbeater.cc:511] Master 127.0.97.62:46615 requested a full tablet report, sending...
I20260923 19:30:11.550354 7985 ts_manager.cc:194] Registered new tserver with Master: c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099)
I20260923 19:30:11.551456 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:30:11.551600 7985 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51539
Sep 23 19:30:11 dist-test-slave-kxfb krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 19:30:11 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:11 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 19:30:11.606261 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:11.594800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.0.97.62:46615)
0923 19:30:11.595115 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.595120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.595179 (+ 59us) server_negotiation.cc:407] Connection header received
0923 19:30:11.595395 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.595399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.595470 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.595569 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:11.597184 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.598017 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.599500 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.599927 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.600029 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.606012 (+ 5983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58780: BlockingRecv error: recv got EOF from 127.0.0.1:58780 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 23 19:30:11 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:30:11.627048 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.611453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.0.97.62:46615)
0923 19:30:11.611629 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.611635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.611652 (+ 17us) server_negotiation.cc:407] Connection header received
0923 19:30:11.611700 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.611706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.611774 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.611862 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:11.613389 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.614482 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.616254 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.616921 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.617027 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.620968 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:11.620972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:11.620976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:11.621005 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:11.624052 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:11.624633 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:11.624638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:11.624642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:11.624770 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:11.625236 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:11.625245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:11.625249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:11.625655 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:11.626036 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.626467 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.626696 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 19:30:11.632254 7985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:30:11.640582 7985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:30:11.669577 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.657411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41091 (local address 127.0.0.1:59138)
0923 19:30:11.658602 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:11.658669 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:11.659195 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:11.660850 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:11.660862 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:11.660896 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:11.661445 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.661466 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.663920 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.663925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.666369 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.666387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.668548 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.668554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.668783 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.668834 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 19:30:11.668976 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 19:30:11.669194 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":1981,"threads_started":1}
I20260923 19:30:11.670881 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.657607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.0.97.2:41091)
0923 19:30:11.659864 (+ 2257us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.659897 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.659929 (+ 32us) server_negotiation.cc:407] Connection header received
0923 19:30:11.660097 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.660105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.660501 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.661647 (+ 1146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:11.661947 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.663709 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.666573 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.668038 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.668172 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.668404 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.670414 (+ 2010us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.670604 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1856,"thread_start_us":1347,"threads_started":1}
I20260923 19:30:11.677856 8263 tablet_service.cc:1467] Processing CreateTablet for tablet 1952ee7bbe05474aa8d1994bd7379384 (DEFAULT_TABLE table=test-table [id=4805154fce784bc8a86c79a1e21f0919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:11.679525 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1952ee7bbe05474aa8d1994bd7379384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:11.682484 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.657518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43099 (local address 127.0.0.1:57370)
0923 19:30:11.659753 (+ 2235us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:11.659777 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:11.666778 (+ 7001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:11.674651 (+ 7873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:11.674657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:11.674686 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:11.675193 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.675209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.676993 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.676999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.679012 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.679026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.681489 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.681496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.681684 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.681728 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 19:30:11.681829 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 19:30:11.681990 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2052,"thread_start_us":832,"threads_started":1}
I20260923 19:30:11.683398 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.658131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.0.97.3:43099)
0923 19:30:11.673630 (+ 15499us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.673637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.673665 (+ 28us) server_negotiation.cc:407] Connection header received
0923 19:30:11.673745 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.673751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.674012 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.674638 (+ 626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:11.675454 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.676823 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.679216 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.680886 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.681027 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.681329 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.682805 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.683013 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15346,"thread_start_us":15961,"threads_started":1}
I20260923 19:30:11.685225 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.659906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35179 (local address 127.0.0.1:53472)
0923 19:30:11.670106 (+ 10200us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:11.670127 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:11.670235 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:11.678417 (+ 8182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:11.678424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:11.678456 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:11.678712 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.678725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.680671 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.680678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.682581 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.682612 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.684408 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.684414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.684629 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.684666 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 19:30:11.684750 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 19:30:11.684915 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10084,"thread_start_us":13617,"threads_started":1}
I20260923 19:30:11.685333 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.660191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.0.97.1:35179)
0923 19:30:11.673934 (+ 13743us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.673945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.673983 (+ 38us) server_negotiation.cc:407] Connection header received
0923 19:30:11.674113 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.674122 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.674447 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.674663 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:11.678932 (+ 4269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.680478 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.682842 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.684244 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.684379 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.684623 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.684840 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.685012 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13518,"thread_start_us":14653,"threads_started":1}
I20260923 19:30:11.687733 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 1952ee7bbe05474aa8d1994bd7379384 (DEFAULT_TABLE table=test-table [id=4805154fce784bc8a86c79a1e21f0919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:11.689239 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1952ee7bbe05474aa8d1994bd7379384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:11.688808 8123 tablet_service.cc:1467] Processing CreateTablet for tablet 1952ee7bbe05474aa8d1994bd7379384 (DEFAULT_TABLE table=test-table [id=4805154fce784bc8a86c79a1e21f0919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:11.690056 8123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1952ee7bbe05474aa8d1994bd7379384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:11.694161 8496 tablet_bootstrap.cc:492] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Bootstrap starting.
I20260923 19:30:11.698576 8496 tablet_bootstrap.cc:654] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:11.700172 8496 log.cc:824] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:11.702893 8498 tablet_bootstrap.cc:492] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: Bootstrap starting.
I20260923 19:30:11.705855 8499 tablet_bootstrap.cc:492] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: Bootstrap starting.
I20260923 19:30:11.706702 8498 tablet_bootstrap.cc:654] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:11.707082 8496 tablet_bootstrap.cc:492] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: No bootstrap required, opened a new log
I20260923 19:30:11.707437 8496 ts_tablet_manager.cc:1397] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260923 19:30:11.708046 8498 log.cc:824] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:11.709115 8499 tablet_bootstrap.cc:654] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:11.710194 8499 log.cc:824] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:11.711836 8498 tablet_bootstrap.cc:492] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: No bootstrap required, opened a new log
I20260923 19:30:11.712146 8498 ts_tablet_manager.cc:1397] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260923 19:30:11.717113 8499 tablet_bootstrap.cc:492] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: No bootstrap required, opened a new log
I20260923 19:30:11.717366 8499 ts_tablet_manager.cc:1397] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260923 19:30:11.717098 8496 raft_consensus.cc:348] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.717621 8496 raft_consensus.cc:374] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:11.717712 8496 raft_consensus.cc:729] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56662f8ce94c4dc49bbc8c4a1cb74b31, State: Initialized, Role: FOLLOWER
I20260923 19:30:11.718415 8496 consensus_queue.cc:260] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.721951 8498 raft_consensus.cc:348] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.722469 8498 raft_consensus.cc:374] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:11.722554 8498 raft_consensus.cc:729] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d4138521c24c8e8b4a9772aa0b6a02, State: Initialized, Role: FOLLOWER
I20260923 19:30:11.723359 8498 consensus_queue.cc:260] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.728811 8499 raft_consensus.cc:348] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.729269 8499 raft_consensus.cc:374] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:11.729375 8499 raft_consensus.cc:729] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aabbf67c1df49269bea3b750e81ebe8, State: Initialized, Role: FOLLOWER
I20260923 19:30:11.730007 8499 consensus_queue.cc:260] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.731297 8333 heartbeater.cc:503] Master 127.0.97.62:46615 was elected leader, sending a full tablet report...
I20260923 19:30:11.732216 8496 ts_tablet_manager.cc:1428] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent starting tablet: real 0.025s user 0.012s sys 0.013s
I20260923 19:30:11.733771 8499 ts_tablet_manager.cc:1428] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260923 19:30:11.734800 8472 heartbeater.cc:503] Master 127.0.97.62:46615 was elected leader, sending a full tablet report...
W20260923 19:30:11.735635 8334 tablet.cc:2369] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:30:11.738067 8498 ts_tablet_manager.cc:1428] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent starting tablet: real 0.026s user 0.018s sys 0.003s
I20260923 19:30:11.773782 8502 raft_consensus.cc:482] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:11.774080 8502 raft_consensus.cc:504] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.775817 8502 leader_election.cc:288] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099), 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179)
I20260923 19:30:11.776474 8504 raft_consensus.cc:482] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:11.776979 8504 raft_consensus.cc:504] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.780764 8504 leader_election.cc:288] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091), c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099)
I20260923 19:30:11.786605 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.776338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41179 (local address 127.0.97.1:35179)
0923 19:30:11.776570 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.776577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.777514 (+ 937us) server_negotiation.cc:407] Connection header received
0923 19:30:11.777861 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.777869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.777989 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.778120 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:11.778773 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.779667 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.784001 (+ 4334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.785468 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.785598 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.785702 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.786162 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.786319 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":70}
I20260923 19:30:11.786645 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.776294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35179 (local address 127.0.97.2:41179)
0923 19:30:11.777387 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:11.777410 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:11.777646 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:11.778175 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:11.778179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:11.778203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:11.778599 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.778608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.779839 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.779843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.782253 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.782270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.785871 (+ 3601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.785877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.786005 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.786028 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 19:30:11.786118 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 19:30:11.786248 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":1,"thread_start_us":457,"threads_started":1}
I20260923 19:30:11.787523 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1952ee7bbe05474aa8d1994bd7379384" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aabbf67c1df49269bea3b750e81ebe8" is_pre_election: true
I20260923 19:30:11.787957 8144 raft_consensus.cc:2454] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 0.
I20260923 19:30:11.788767 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.777159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47303 (local address 127.0.97.3:43099)
0923 19:30:11.777394 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.777401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.777571 (+ 170us) server_negotiation.cc:407] Connection header received
0923 19:30:11.777840 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.777844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.777902 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.777972 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:11.778706 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.779818 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.785742 (+ 5924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.787153 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.787283 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.787422 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.788298 (+ 876us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.788489 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":57}
I20260923 19:30:11.788648 8213 leader_election.cc:302] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56662f8ce94c4dc49bbc8c4a1cb74b31, 6aabbf67c1df49269bea3b750e81ebe8; no voters:
I20260923 19:30:11.789302 8502 raft_consensus.cc:2781] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:30:11.789451 8502 raft_consensus.cc:482] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:30:11.789549 8502 raft_consensus.cc:3037] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:11.789606 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.776436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43099 (local address 127.0.97.2:47303)
0923 19:30:11.777469 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:11.777491 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:11.777666 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:11.778008 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:11.778012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:11.778036 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:11.778487 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.778501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.781091 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.781098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.785538 (+ 4440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.785554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.787849 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.787857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.788054 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.788094 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 19:30:11.788238 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 19:30:11.788423 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":1467,"spinlock_wait_cycles":9472,"thread_start_us":943,"threads_started":1}
I20260923 19:30:11.791049 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1952ee7bbe05474aa8d1994bd7379384" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" is_pre_election: true
I20260923 19:30:11.791373 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.780652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38145 (local address 127.0.97.2:41091)
0923 19:30:11.780828 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.780833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.780948 (+ 115us) server_negotiation.cc:407] Connection header received
0923 19:30:11.781239 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.781246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.781338 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.781418 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:11.782392 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.783494 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.788813 (+ 5319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.790121 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.790261 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.790386 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.790967 (+ 581us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.791115 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 19:30:11.791792 8427 raft_consensus.cc:2454] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 0.
I20260923 19:30:11.792135 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.779692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41091 (local address 127.0.97.1:38145)
0923 19:30:11.780818 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:11.780842 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:11.781068 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:11.781626 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:11.781631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:11.781658 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:11.782160 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.782175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.783693 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.783698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.787133 (+ 3435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.787146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.790529 (+ 3383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.790535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.790779 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.790816 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 19:30:11.791576 (+ 760us) client_negotiation.cc:240] Negotiation successful
0923 19:30:11.791772 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":95,"thread_start_us":3550,"threads_started":1}
I20260923 19:30:11.792755 8502 raft_consensus.cc:504] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.792977 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1952ee7bbe05474aa8d1994bd7379384" candidate_uuid: "6aabbf67c1df49269bea3b750e81ebe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" is_pre_election: true
I20260923 19:30:11.793363 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.780030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43099 (local address 127.0.97.1:53909)
0923 19:30:11.783695 (+ 3665us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:11.783714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:11.783855 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:11.784591 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:11.784596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:11.784623 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:11.786503 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.786519 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.788246 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.788252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.790180 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:11.790194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.792509 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.792516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:11.792701 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.792738 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 19:30:11.792838 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 19:30:11.792982 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3479,"thread_start_us":412,"threads_started":1}
I20260923 19:30:11.793499 8283 raft_consensus.cc:2379] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6aabbf67c1df49269bea3b750e81ebe8 in current term 1: Already voted for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in this term.
I20260923 19:30:11.793988 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.780243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53909 (local address 127.0.97.3:43099)
0923 19:30:11.782987 (+ 2744us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.783021 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.783981 (+ 960us) server_negotiation.cc:407] Connection header received
0923 19:30:11.784058 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.784064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.784149 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.784291 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:11.786744 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.788037 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.790400 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.791932 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.792148 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.792292 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.793333 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.793633 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2532,"thread_start_us":2231,"threads_started":1}
I20260923 19:30:11.794420 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1952ee7bbe05474aa8d1994bd7379384" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02"
I20260923 19:30:11.794639 8427 raft_consensus.cc:3037] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:11.794730 8502 leader_election.cc:288] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 election: Requested vote from peers c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099), 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179)
I20260923 19:30:11.794744 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1952ee7bbe05474aa8d1994bd7379384" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aabbf67c1df49269bea3b750e81ebe8"
I20260923 19:30:11.794915 8144 raft_consensus.cc:3037] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:11.794770 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1952ee7bbe05474aa8d1994bd7379384" candidate_uuid: "6aabbf67c1df49269bea3b750e81ebe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" is_pre_election: true
I20260923 19:30:11.795888 8075 leader_election.cc:302] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6aabbf67c1df49269bea3b750e81ebe8; no voters: 56662f8ce94c4dc49bbc8c4a1cb74b31, c4d4138521c24c8e8b4a9772aa0b6a02
W20260923 19:30:11.796790 8473 tablet.cc:2369] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:30:11.797914 8144 raft_consensus.cc:2454] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 1.
I20260923 19:30:11.798038 8427 raft_consensus.cc:2454] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 1.
I20260923 19:30:11.798087 8504 raft_consensus.cc:2726] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 19:30:11.798398 8213 leader_election.cc:302] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56662f8ce94c4dc49bbc8c4a1cb74b31, 6aabbf67c1df49269bea3b750e81ebe8; no voters:
I20260923 19:30:11.798736 8502 raft_consensus.cc:2781] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:11.799242 8502 raft_consensus.cc:686] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 LEADER]: Becoming Leader. State: Replica: 56662f8ce94c4dc49bbc8c4a1cb74b31, State: Running, Role: LEADER
I20260923 19:30:11.799811 8502 consensus_queue.cc:237] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } }
I20260923 19:30:11.804875 7984 catalog_manager.cc:5495] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } health_report { overall_health: UNKNOWN } } }
I20260923 19:30:11.828248 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:11.822237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.0.97.2:41091)
0923 19:30:11.822464 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:11.822472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:11.822605 (+ 133us) server_negotiation.cc:407] Connection header received
0923 19:30:11.823048 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:11.823052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:11.823182 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:11.823339 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:11.823743 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.824576 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.825923 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:11.826384 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:11.826494 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:11.826931 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:11.827592 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:11.827726 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:11.827828 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 19:30:11.827924 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 19:30:11.837574 8427 raft_consensus.cc:1261] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Refusing update from remote peer 56662f8ce94c4dc49bbc8c4a1cb74b31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:11.837635 8144 raft_consensus.cc:1261] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Refusing update from remote peer 56662f8ce94c4dc49bbc8c4a1cb74b31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:11.838727 8510 consensus_queue.cc:1035] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:11.839141 8502 consensus_queue.cc:1035] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:11.856555 8514 mvcc.cc:205] Tried to move back new op lower bound from 7332625661275684864 to 7332625661136265216. Current Snapshot: MvccSnapshot[applied={T|T < 7332625661275684864}]
I20260923 19:30:11.867983 7985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:11.868379 7985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:11.872491 7985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a: Sending DeleteTablet for 3 replicas of tablet 1952ee7bbe05474aa8d1994bd7379384
I20260923 19:30:11.873766 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1952ee7bbe05474aa8d1994bd7379384 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
I20260923 19:30:11.874382 8123 tablet_service.cc:1514] Processing DeleteTablet for tablet 1952ee7bbe05474aa8d1994bd7379384 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53472
I20260923 19:30:11.873767 8263 tablet_service.cc:1514] Processing DeleteTablet for tablet 1952ee7bbe05474aa8d1994bd7379384 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59138
I20260923 19:30:11.875170 7984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:30:11.875952 7984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:30:11.883672 8523 tablet_replica.cc:330] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: stopping tablet replica
I20260923 19:30:11.884182 8523 raft_consensus.cc:2229] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:11.884788 8523 raft_consensus.cc:2258] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:11.885268 8401 tablet_service.cc:1467] Processing CreateTablet for tablet eb58431c63804fc780a5eb3962156516 (DEFAULT_TABLE table=test-table [id=e8908cb38162446e87a444ead9b146e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:11.885788 8262 tablet_service.cc:1467] Processing CreateTablet for tablet eb58431c63804fc780a5eb3962156516 (DEFAULT_TABLE table=test-table [id=e8908cb38162446e87a444ead9b146e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:11.885936 8401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb58431c63804fc780a5eb3962156516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:11.886811 8122 tablet_service.cc:1467] Processing CreateTablet for tablet eb58431c63804fc780a5eb3962156516 (DEFAULT_TABLE table=test-table [id=e8908cb38162446e87a444ead9b146e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:11.887456 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb58431c63804fc780a5eb3962156516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:11.887485 8262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb58431c63804fc780a5eb3962156516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:11.892627 8498 tablet_bootstrap.cc:492] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: Bootstrap starting.
I20260923 19:30:11.894556 8498 tablet_bootstrap.cc:654] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:11.896085 8496 tablet_bootstrap.cc:492] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Bootstrap starting.
I20260923 19:30:11.896549 8523 ts_tablet_manager.cc:1905] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:11.898051 8496 tablet_bootstrap.cc:654] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:11.903929 8498 tablet_bootstrap.cc:492] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: No bootstrap required, opened a new log
I20260923 19:30:11.904145 8498 ts_tablet_manager.cc:1397] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20260923 19:30:11.904445 8525 tablet_replica.cc:330] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: stopping tablet replica
I20260923 19:30:11.904973 8525 raft_consensus.cc:2229] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:30:11.905074 8498 raft_consensus.cc:348] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:11.905319 8498 raft_consensus.cc:374] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:11.905414 8498 raft_consensus.cc:729] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d4138521c24c8e8b4a9772aa0b6a02, State: Initialized, Role: FOLLOWER
I20260923 19:30:11.905722 8525 raft_consensus.cc:2258] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:11.905699 8498 consensus_queue.cc:260] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:11.906265 8498 ts_tablet_manager.cc:1428] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 19:30:11.907845 8525 ts_tablet_manager.cc:1905] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:11.908746 8524 tablet_replica.cc:330] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: stopping tablet replica
I20260923 19:30:11.909281 8524 raft_consensus.cc:2229] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:11.903896 8523 ts_tablet_manager.cc:1918] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:11.910696 8523 log.cc:1196] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/1952ee7bbe05474aa8d1994bd7379384
I20260923 19:30:11.911407 8523 ts_tablet_manager.cc:1939] T 1952ee7bbe05474aa8d1994bd7379384 P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting consensus metadata
I20260923 19:30:11.911473 8524 raft_consensus.cc:2258] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:11.912287 8499 tablet_bootstrap.cc:492] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: Bootstrap starting.
I20260923 19:30:11.914206 8499 tablet_bootstrap.cc:654] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:11.914605 7972 catalog_manager.cc:4841] TS c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099): tablet 1952ee7bbe05474aa8d1994bd7379384 (table test-table [id=4805154fce784bc8a86c79a1e21f0919]) successfully deleted
I20260923 19:30:11.914742 8524 ts_tablet_manager.cc:1905] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:11.917359 8499 tablet_bootstrap.cc:492] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: No bootstrap required, opened a new log
I20260923 19:30:11.917511 8499 ts_tablet_manager.cc:1397] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 19:30:11.918352 8499 raft_consensus.cc:348] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:11.918617 8499 raft_consensus.cc:374] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:11.918712 8499 raft_consensus.cc:729] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aabbf67c1df49269bea3b750e81ebe8, State: Initialized, Role: FOLLOWER
I20260923 19:30:11.919054 8499 consensus_queue.cc:260] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:11.920678 8496 tablet_bootstrap.cc:492] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: No bootstrap required, opened a new log
I20260923 19:30:11.920874 8496 ts_tablet_manager.cc:1397] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.002s
I20260923 19:30:11.921922 8496 raft_consensus.cc:348] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:11.922188 8496 raft_consensus.cc:374] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:11.922338 8496 raft_consensus.cc:729] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56662f8ce94c4dc49bbc8c4a1cb74b31, State: Initialized, Role: FOLLOWER
I20260923 19:30:11.922658 8496 consensus_queue.cc:260] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:11.923062 8499 ts_tablet_manager.cc:1428] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260923 19:30:11.923719 8525 ts_tablet_manager.cc:1918] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:11.923938 8525 log.cc:1196] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/1952ee7bbe05474aa8d1994bd7379384
I20260923 19:30:11.924536 8525 ts_tablet_manager.cc:1939] T 1952ee7bbe05474aa8d1994bd7379384 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting consensus metadata
I20260923 19:30:11.925446 8496 ts_tablet_manager.cc:1428] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260923 19:30:11.926532 7970 catalog_manager.cc:4841] TS 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091): tablet 1952ee7bbe05474aa8d1994bd7379384 (table test-table [id=4805154fce784bc8a86c79a1e21f0919]) successfully deleted
I20260923 19:30:11.929375 8524 ts_tablet_manager.cc:1918] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:11.929519 8524 log.cc:1196] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/1952ee7bbe05474aa8d1994bd7379384
I20260923 19:30:11.929934 8524 ts_tablet_manager.cc:1939] T 1952ee7bbe05474aa8d1994bd7379384 P 6aabbf67c1df49269bea3b750e81ebe8: Deleting consensus metadata
I20260923 19:30:11.931516 7970 catalog_manager.cc:4841] TS 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179): tablet 1952ee7bbe05474aa8d1994bd7379384 (table test-table [id=4805154fce784bc8a86c79a1e21f0919]) successfully deleted
I20260923 19:30:12.032505 8502 raft_consensus.cc:482] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:12.032780 8502 raft_consensus.cc:504] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.033564 8502 leader_election.cc:288] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179), c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099)
I20260923 19:30:12.034264 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb58431c63804fc780a5eb3962156516" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aabbf67c1df49269bea3b750e81ebe8" is_pre_election: true
I20260923 19:30:12.034319 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb58431c63804fc780a5eb3962156516" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" is_pre_election: true
I20260923 19:30:12.034583 8144 raft_consensus.cc:2454] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 0.
I20260923 19:30:12.034636 8427 raft_consensus.cc:2454] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 0.
I20260923 19:30:12.035130 8215 leader_election.cc:302] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56662f8ce94c4dc49bbc8c4a1cb74b31, c4d4138521c24c8e8b4a9772aa0b6a02; no voters:
I20260923 19:30:12.035446 8502 raft_consensus.cc:2781] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:30:12.035575 8502 raft_consensus.cc:482] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:30:12.035686 8502 raft_consensus.cc:3037] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.038198 8502 raft_consensus.cc:504] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.038877 8502 leader_election.cc:288] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 election: Requested vote from peers 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179), c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099)
I20260923 19:30:12.039603 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb58431c63804fc780a5eb3962156516" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aabbf67c1df49269bea3b750e81ebe8"
I20260923 19:30:12.039719 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb58431c63804fc780a5eb3962156516" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02"
I20260923 19:30:12.039844 8144 raft_consensus.cc:3037] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.039984 8427 raft_consensus.cc:3037] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.042428 8144 raft_consensus.cc:2454] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 1.
I20260923 19:30:12.042987 8213 leader_election.cc:302] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56662f8ce94c4dc49bbc8c4a1cb74b31, 6aabbf67c1df49269bea3b750e81ebe8; no voters:
I20260923 19:30:12.043013 8427 raft_consensus.cc:2454] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 1.
I20260923 19:30:12.043304 8502 raft_consensus.cc:2781] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:12.043524 8502 raft_consensus.cc:686] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 LEADER]: Becoming Leader. State: Replica: 56662f8ce94c4dc49bbc8c4a1cb74b31, State: Running, Role: LEADER
I20260923 19:30:12.043900 8502 consensus_queue.cc:237] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.047173 7985 catalog_manager.cc:5495] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } health_report { overall_health: UNKNOWN } } }
Sep 23 19:30:12 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:30:12.124642 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.097239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46615 (local address 127.0.0.1:58792)
0923 19:30:12.098208 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:12.098229 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:12.098440 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:12.098907 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:12.098913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:12.099665 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:12.099993 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.100002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.101391 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.101395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.102521 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.102533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.103493 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.103498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.103631 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.114692 (+ 11061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:12.114720 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:12.117633 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:12.121602 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:12.121615 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:12.121622 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:12.122151 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:12.122681 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:12.122687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:12.122691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:12.122884 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:12.123596 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:12.123603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:12.123807 (+ 204us) client_negotiation.cc:769] Sending connection context
0923 19:30:12.123983 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 19:30:12.124269 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":661,"threads_started":1}
I20260923 19:30:12.124953 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.097301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.0.97.62:46615)
0923 19:30:12.097700 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.097708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.098377 (+ 669us) server_negotiation.cc:407] Connection header received
0923 19:30:12.098593 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.098599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.098691 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.098852 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:12.100136 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.101238 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.102655 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.103293 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.103423 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.117934 (+ 14511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:12.117942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:12.117947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:12.118003 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:12.121410 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:12.122331 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:12.122337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:12.122340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:12.122503 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:12.123024 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:12.123029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:12.123032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:12.123435 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:12.123603 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.124136 (+ 533us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.124421 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":228}
I20260923 19:30:12.131197 7985 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58792:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 19:30:12.132550 7985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:30:12.140568 8122 tablet_service.cc:1467] Processing CreateTablet for tablet d21c5c62bed347e0a59731ece908f726 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06ba69adce4d4be5be85d9d8ebc85d83]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 19:30:12.141219 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21c5c62bed347e0a59731ece908f726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:12.141021 8403 tablet_service.cc:1467] Processing CreateTablet for tablet d21c5c62bed347e0a59731ece908f726 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06ba69adce4d4be5be85d9d8ebc85d83]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 19:30:12.141198 8263 tablet_service.cc:1467] Processing CreateTablet for tablet d21c5c62bed347e0a59731ece908f726 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06ba69adce4d4be5be85d9d8ebc85d83]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 19:30:12.141568 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21c5c62bed347e0a59731ece908f726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:12.141845 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21c5c62bed347e0a59731ece908f726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:12.148912 8499 tablet_bootstrap.cc:492] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: Bootstrap starting.
I20260923 19:30:12.149036 8498 tablet_bootstrap.cc:492] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: Bootstrap starting.
I20260923 19:30:12.150509 8499 tablet_bootstrap.cc:654] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:12.151080 8498 tablet_bootstrap.cc:654] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:12.152544 8499 tablet_bootstrap.cc:492] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: No bootstrap required, opened a new log
I20260923 19:30:12.152699 8499 ts_tablet_manager.cc:1397] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260923 19:30:12.153297 8499 raft_consensus.cc:348] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.153509 8499 raft_consensus.cc:374] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:12.153587 8499 raft_consensus.cc:729] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aabbf67c1df49269bea3b750e81ebe8, State: Initialized, Role: FOLLOWER
I20260923 19:30:12.153836 8499 consensus_queue.cc:260] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.154511 8504 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.154822 8504 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:30:12.155289 8496 tablet_bootstrap.cc:492] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Bootstrap starting.
I20260923 19:30:12.155465 8499 ts_tablet_manager.cc:1428] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260923 19:30:12.157009 8496 tablet_bootstrap.cc:654] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:12.163906 8498 tablet_bootstrap.cc:492] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: No bootstrap required, opened a new log
I20260923 19:30:12.164106 8498 ts_tablet_manager.cc:1397] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260923 19:30:12.164642 8496 tablet_bootstrap.cc:492] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: No bootstrap required, opened a new log
I20260923 19:30:12.164846 8496 ts_tablet_manager.cc:1397] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260923 19:30:12.165001 8498 raft_consensus.cc:348] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.165269 8498 raft_consensus.cc:374] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:12.165359 8498 raft_consensus.cc:729] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d4138521c24c8e8b4a9772aa0b6a02, State: Initialized, Role: FOLLOWER
I20260923 19:30:12.165709 8498 consensus_queue.cc:260] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.165695 8496 raft_consensus.cc:348] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.165980 8496 raft_consensus.cc:374] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:12.166075 8496 raft_consensus.cc:729] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56662f8ce94c4dc49bbc8c4a1cb74b31, State: Initialized, Role: FOLLOWER
I20260923 19:30:12.166162 8503 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.166356 8496 consensus_queue.cc:260] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.166499 8503 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:30:12.167112 8502 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.167449 8502 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:30:12.168167 8498 ts_tablet_manager.cc:1428] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260923 19:30:12.169112 8496 ts_tablet_manager.cc:1428] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
W20260923 19:30:12.170404 8194 tablet.cc:2369] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:30:12.312424 8503 raft_consensus.cc:482] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:12.312719 8503 raft_consensus.cc:504] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.314512 8503 leader_election.cc:288] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179), 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091)
I20260923 19:30:12.323175 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.314979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59899 (local address 127.0.97.1:35179)
0923 19:30:12.316146 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.316150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.316166 (+ 16us) server_negotiation.cc:407] Connection header received
0923 19:30:12.316232 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.316236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.316293 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.316376 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:12.316818 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.317944 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.320698 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.321999 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.322393 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.322495 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.322667 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.322809 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":914,"threads_started":1}
I20260923 19:30:12.323236 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.315632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32979 (local address 127.0.97.2:41091)
0923 19:30:12.315833 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.315840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.316304 (+ 464us) server_negotiation.cc:407] Connection header received
0923 19:30:12.316549 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.316554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.316626 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.316701 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:12.317070 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.318073 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.320698 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.321978 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.322119 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.322228 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.322783 (+ 555us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.322965 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 19:30:12.323380 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.314817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35179 (local address 127.0.97.3:59899)
0923 19:30:12.315453 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:12.315486 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:12.315657 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:12.316427 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:12.316432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:12.316465 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:12.316663 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.316671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.318082 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.318086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.320507 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.320522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.322185 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.322192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.322484 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.322544 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 19:30:12.322635 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 19:30:12.322886 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":130,"thread_start_us":336,"threads_started":1}
I20260923 19:30:12.323563 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.315507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41091 (local address 127.0.97.3:32979)
0923 19:30:12.316205 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:12.316224 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:12.316362 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:12.316744 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:12.316748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:12.316770 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:12.316939 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.316947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.318202 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.318206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.320499 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.320514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.322268 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.322274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.322585 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.322616 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 19:30:12.322712 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 19:30:12.322851 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":205,"thread_start_us":461,"threads_started":1}
I20260923 19:30:12.324141 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21c5c62bed347e0a59731ece908f726" candidate_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" is_pre_election: true
I20260923 19:30:12.324240 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21c5c62bed347e0a59731ece908f726" candidate_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aabbf67c1df49269bea3b750e81ebe8" is_pre_election: true
I20260923 19:30:12.324448 8283 raft_consensus.cc:2454] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d4138521c24c8e8b4a9772aa0b6a02 in term 0.
I20260923 19:30:12.324556 8144 raft_consensus.cc:2454] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d4138521c24c8e8b4a9772aa0b6a02 in term 0.
I20260923 19:30:12.324966 8353 leader_election.cc:302] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56662f8ce94c4dc49bbc8c4a1cb74b31, c4d4138521c24c8e8b4a9772aa0b6a02; no voters:
I20260923 19:30:12.325371 8503 raft_consensus.cc:2781] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:30:12.325496 8503 raft_consensus.cc:482] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:30:12.325595 8503 raft_consensus.cc:3037] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.327955 8503 raft_consensus.cc:504] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.328539 8503 leader_election.cc:288] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [CANDIDATE]: Term 1 election: Requested vote from peers 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179), 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091)
I20260923 19:30:12.329046 8144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21c5c62bed347e0a59731ece908f726" candidate_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aabbf67c1df49269bea3b750e81ebe8"
I20260923 19:30:12.329151 8283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21c5c62bed347e0a59731ece908f726" candidate_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31"
I20260923 19:30:12.329242 8144 raft_consensus.cc:3037] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.329360 8283 raft_consensus.cc:3037] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.331409 8144 raft_consensus.cc:2454] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d4138521c24c8e8b4a9772aa0b6a02 in term 1.
I20260923 19:30:12.331614 8283 raft_consensus.cc:2454] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d4138521c24c8e8b4a9772aa0b6a02 in term 1.
I20260923 19:30:12.331846 8353 leader_election.cc:302] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aabbf67c1df49269bea3b750e81ebe8, c4d4138521c24c8e8b4a9772aa0b6a02; no voters:
I20260923 19:30:12.332154 8503 raft_consensus.cc:2781] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:12.332561 8503 raft_consensus.cc:686] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 LEADER]: Becoming Leader. State: Replica: c4d4138521c24c8e8b4a9772aa0b6a02, State: Running, Role: LEADER
I20260923 19:30:12.333015 8503 consensus_queue.cc:237] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.335054 8542 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: TxnStatusTablet state changed. Reason: New leader c4d4138521c24c8e8b4a9772aa0b6a02. Latest consensus state: current_term: 1 leader_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.335451 8542 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: This TxnStatusTablet replica's current role is: LEADER
I20260923 19:30:12.336349 8544 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 19:30:12.336575 8544 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 19:30:12.336887 7984 catalog_manager.cc:5495] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } health_report { overall_health: HEALTHY } } }
I20260923 19:30:12.368530 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.360811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43099 (local address 127.0.0.1:57374)
0923 19:30:12.361078 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:12.361098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:12.361253 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:12.362338 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:12.362343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:12.362358 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:30:12.362615 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.362628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.364322 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.364326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.365448 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.365459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.366573 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.366578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.366723 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.366847 (+ 124us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.367985 (+ 1138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.368013 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 19:30:12.368090 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 19:30:12.368235 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 19:30:12.368613 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.360974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.0.97.3:43099)
0923 19:30:12.361839 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.361848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.361874 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:30:12.361997 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.362002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.362111 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.362328 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:12.362801 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.364129 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.365615 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.366408 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.366549 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.367056 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.367850 (+ 794us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.368007 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.368150 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.368240 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":534,"threads_started":1}
I20260923 19:30:12.374229 8144 raft_consensus.cc:1261] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Refusing update from remote peer c4d4138521c24c8e8b4a9772aa0b6a02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:12.374320 8283 raft_consensus.cc:1261] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Refusing update from remote peer c4d4138521c24c8e8b4a9772aa0b6a02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:12.375236 8542 consensus_queue.cc:1035] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:12.375584 8503 consensus_queue.cc:1035] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:12.379235 8504 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: TxnStatusTablet state changed. Reason: New leader c4d4138521c24c8e8b4a9772aa0b6a02. Latest consensus state: current_term: 1 leader_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.379608 8504 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:30:12.381119 8502 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: TxnStatusTablet state changed. Reason: New leader c4d4138521c24c8e8b4a9772aa0b6a02. Latest consensus state: current_term: 1 leader_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.381415 8502 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:30:12.383883 8548 mvcc.cc:205] Tried to move back new op lower bound from 7332625663476301824 to 7332625663319977984. Current Snapshot: MvccSnapshot[applied={T|T < 7332625663476301824}]
I20260923 19:30:12.384327 8542 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.384604 8542 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: This TxnStatusTablet replica's current role is: LEADER
I20260923 19:30:12.384825 8542 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.385064 8542 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P c4d4138521c24c8e8b4a9772aa0b6a02: This TxnStatusTablet replica's current role is: LEADER
I20260923 19:30:12.390892 8550 mvcc.cc:205] Tried to move back new op lower bound from 7332625663476301824 to 7332625663319977984. Current Snapshot: MvccSnapshot[applied={T|T < 7332625663476301824}]
I20260923 19:30:12.391324 8504 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.391634 8504 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P 6aabbf67c1df49269bea3b750e81ebe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:30:12.395042 8502 tablet_replica.cc:439] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } } }
I20260923 19:30:12.395316 8502 tablet_replica.cc:442] T d21c5c62bed347e0a59731ece908f726 P 56662f8ce94c4dc49bbc8c4a1cb74b31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 19:30:12.417060 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.408379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41693 (local address 127.0.97.3:43099)
0923 19:30:12.408612 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.408619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.410691 (+ 2072us) server_negotiation.cc:407] Connection header received
0923 19:30:12.410928 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.410932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.411042 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.411136 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:12.411697 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.412475 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.414628 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.415326 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.415451 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.416302 (+ 851us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.416432 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.416537 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.416680 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.416758 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 19:30:12.417528 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.408288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43099 (local address 127.0.97.2:41693)
0923 19:30:12.410521 (+ 2233us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:12.410551 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:12.410760 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:12.411177 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:12.411183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:12.411208 (+ 25us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:30:12.411519 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.411531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.412606 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.412611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.414424 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.414438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.415488 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.415494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.415679 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.415763 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.416546 (+ 783us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.416558 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 19:30:12.416647 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 19:30:12.416801 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2015,"thread_start_us":1834,"threads_started":1}
I20260923 19:30:12.430181 8427 raft_consensus.cc:1261] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Refusing update from remote peer 56662f8ce94c4dc49bbc8c4a1cb74b31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:12.430400 8144 raft_consensus.cc:1261] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Refusing update from remote peer 56662f8ce94c4dc49bbc8c4a1cb74b31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:12.430845 8502 consensus_queue.cc:1035] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:12.431288 8502 consensus_queue.cc:1035] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:12.482445 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.475368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55855 (local address 127.0.97.2:41091)
0923 19:30:12.475535 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.475551 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.475844 (+ 293us) server_negotiation.cc:407] Connection header received
0923 19:30:12.476056 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.476062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.476167 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.476282 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:12.477110 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.477907 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.480221 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.480674 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.480777 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.481391 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.481533 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.481678 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.481924 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.482049 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 19:30:12.482532 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.475077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41091 (local address 127.0.97.3:55855)
0923 19:30:12.475716 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:12.475738 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:12.475893 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:12.476478 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:12.476483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:12.476498 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:30:12.476911 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.476924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.478054 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.478059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.480050 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.480066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.480863 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.480870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.481074 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.481168 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.481695 (+ 527us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.481711 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 19:30:12.481854 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 19:30:12.482065 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":1138,"threads_started":1}
I20260923 19:30:12.514797 7984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:12.515007 7984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:12.518160 7984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a: Sending DeleteTablet for 3 replicas of tablet eb58431c63804fc780a5eb3962156516
I20260923 19:30:12.519119 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet eb58431c63804fc780a5eb3962156516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53472
I20260923 19:30:12.520813 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet eb58431c63804fc780a5eb3962156516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
I20260923 19:30:12.520815 8263 tablet_service.cc:1514] Processing DeleteTablet for tablet eb58431c63804fc780a5eb3962156516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59138
I20260923 19:30:12.522949 8567 tablet_replica.cc:330] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: stopping tablet replica
I20260923 19:30:12.523200 8567 raft_consensus.cc:2229] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:30:12.523687 8567 raft_consensus.cc:2258] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:12.524333 8568 tablet_replica.cc:330] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: stopping tablet replica
I20260923 19:30:12.524497 8566 tablet_replica.cc:330] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: stopping tablet replica
I20260923 19:30:12.524549 8568 raft_consensus.cc:2229] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:12.524710 8566 raft_consensus.cc:2229] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:12.524856 8568 raft_consensus.cc:2258] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:12.524919 8566 raft_consensus.cc:2258] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:12.525931 8566 ts_tablet_manager.cc:1905] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:12.527750 8568 ts_tablet_manager.cc:1905] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:12.531010 8567 ts_tablet_manager.cc:1905] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:12.532456 8566 ts_tablet_manager.cc:1918] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:30:12.532583 8566 log.cc:1196] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/eb58431c63804fc780a5eb3962156516
I20260923 19:30:12.533071 8566 ts_tablet_manager.cc:1939] T eb58431c63804fc780a5eb3962156516 P 6aabbf67c1df49269bea3b750e81ebe8: Deleting consensus metadata
I20260923 19:30:12.534128 7970 catalog_manager.cc:4841] TS 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179): tablet eb58431c63804fc780a5eb3962156516 (table test-table [id=e8908cb38162446e87a444ead9b146e3]) successfully deleted
I20260923 19:30:12.537387 8567 ts_tablet_manager.cc:1918] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:30:12.537436 8568 ts_tablet_manager.cc:1918] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:30:12.537499 8567 log.cc:1196] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/eb58431c63804fc780a5eb3962156516
I20260923 19:30:12.537596 8568 log.cc:1196] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/eb58431c63804fc780a5eb3962156516
I20260923 19:30:12.537937 8567 ts_tablet_manager.cc:1939] T eb58431c63804fc780a5eb3962156516 P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting consensus metadata
I20260923 19:30:12.538113 8568 ts_tablet_manager.cc:1939] T eb58431c63804fc780a5eb3962156516 P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting consensus metadata
I20260923 19:30:12.538940 7970 catalog_manager.cc:4841] TS 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091): tablet eb58431c63804fc780a5eb3962156516 (table test-table [id=e8908cb38162446e87a444ead9b146e3]) successfully deleted
I20260923 19:30:12.539281 7972 catalog_manager.cc:4841] TS c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099): tablet eb58431c63804fc780a5eb3962156516 (table test-table [id=e8908cb38162446e87a444ead9b146e3]) successfully deleted
Sep 23 19:30:12 dist-test-slave-kxfb krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 19:30:12 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:12 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 19:30:12.595927 8221 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:12.583594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.0.97.62:46615)
0923 19:30:12.583840 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.583848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.584302 (+ 454us) server_negotiation.cc:407] Connection header received
0923 19:30:12.584601 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.584606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.584699 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.584804 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:12.586676 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.587416 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.589173 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.589574 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.589670 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.595671 (+ 6001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58794: BlockingRecv error: recv got EOF from 127.0.0.1:58794 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 19:30:12 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 19:30:12.615084 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.599174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.0.97.62:46615)
0923 19:30:12.599387 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.599395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.599620 (+ 225us) server_negotiation.cc:407] Connection header received
0923 19:30:12.599784 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.599791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.599881 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.600016 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:12.601142 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.602330 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.603603 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.604301 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.604437 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.608424 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:12.608431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:12.608437 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:12.608487 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:12.612061 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:12.612782 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:12.612787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:12.612791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:12.612982 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:12.613358 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:12.613363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:12.613366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:12.613767 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:12.614044 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.614390 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.614654 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 19:30:12.619199 7984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:30:12.619872 7984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:30:12.626986 8263 tablet_service.cc:1467] Processing CreateTablet for tablet af9ecbfef70f47aa96698f087a7a65ca (DEFAULT_TABLE table=test-table [id=2c16a8d00a9644e39067ef665c4b010d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:12.627271 8122 tablet_service.cc:1467] Processing CreateTablet for tablet af9ecbfef70f47aa96698f087a7a65ca (DEFAULT_TABLE table=test-table [id=2c16a8d00a9644e39067ef665c4b010d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:12.627410 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9ecbfef70f47aa96698f087a7a65ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:12.627450 8403 tablet_service.cc:1467] Processing CreateTablet for tablet af9ecbfef70f47aa96698f087a7a65ca (DEFAULT_TABLE table=test-table [id=2c16a8d00a9644e39067ef665c4b010d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:12.627712 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9ecbfef70f47aa96698f087a7a65ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:12.628001 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9ecbfef70f47aa96698f087a7a65ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:12.633641 8496 tablet_bootstrap.cc:492] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: Bootstrap starting.
I20260923 19:30:12.633973 8499 tablet_bootstrap.cc:492] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: Bootstrap starting.
I20260923 19:30:12.635298 8496 tablet_bootstrap.cc:654] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:12.635491 8499 tablet_bootstrap.cc:654] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:12.635648 8498 tablet_bootstrap.cc:492] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: Bootstrap starting.
I20260923 19:30:12.637602 8498 tablet_bootstrap.cc:654] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:12.645171 8499 tablet_bootstrap.cc:492] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: No bootstrap required, opened a new log
I20260923 19:30:12.645318 8499 ts_tablet_manager.cc:1397] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260923 19:30:12.645542 8498 tablet_bootstrap.cc:492] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: No bootstrap required, opened a new log
I20260923 19:30:12.645727 8498 ts_tablet_manager.cc:1397] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260923 19:30:12.646091 8499 raft_consensus.cc:348] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.646413 8499 raft_consensus.cc:374] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:12.646505 8499 raft_consensus.cc:729] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aabbf67c1df49269bea3b750e81ebe8, State: Initialized, Role: FOLLOWER
I20260923 19:30:12.646746 8496 tablet_bootstrap.cc:492] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: No bootstrap required, opened a new log
I20260923 19:30:12.646749 8499 consensus_queue.cc:260] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.646925 8496 ts_tablet_manager.cc:1397] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260923 19:30:12.646938 8498 raft_consensus.cc:348] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.647239 8498 raft_consensus.cc:374] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:12.647344 8498 raft_consensus.cc:729] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d4138521c24c8e8b4a9772aa0b6a02, State: Initialized, Role: FOLLOWER
I20260923 19:30:12.647621 8498 consensus_queue.cc:260] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.647825 8499 ts_tablet_manager.cc:1428] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 19:30:12.648622 8498 ts_tablet_manager.cc:1428] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 19:30:12.649698 8496 raft_consensus.cc:348] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.650004 8496 raft_consensus.cc:374] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:12.650146 8496 raft_consensus.cc:729] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56662f8ce94c4dc49bbc8c4a1cb74b31, State: Initialized, Role: FOLLOWER
I20260923 19:30:12.650593 8496 consensus_queue.cc:260] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.651304 8496 ts_tablet_manager.cc:1428] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 19:30:12.665928 8502 raft_consensus.cc:482] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:12.666198 8502 raft_consensus.cc:504] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.666858 8502 leader_election.cc:288] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179), c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099)
I20260923 19:30:12.668344 8143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9ecbfef70f47aa96698f087a7a65ca" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aabbf67c1df49269bea3b750e81ebe8" is_pre_election: true
I20260923 19:30:12.668385 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9ecbfef70f47aa96698f087a7a65ca" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" is_pre_election: true
I20260923 19:30:12.668620 8143 raft_consensus.cc:2454] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 0.
I20260923 19:30:12.668720 8427 raft_consensus.cc:2454] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 0.
I20260923 19:30:12.669305 8213 leader_election.cc:302] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56662f8ce94c4dc49bbc8c4a1cb74b31, 6aabbf67c1df49269bea3b750e81ebe8; no voters:
I20260923 19:30:12.670468 8502 raft_consensus.cc:2781] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:30:12.670598 8502 raft_consensus.cc:482] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:30:12.671306 8502 raft_consensus.cc:3037] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.674835 8502 raft_consensus.cc:504] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.675829 8502 leader_election.cc:288] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 election: Requested vote from peers 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179), c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099)
I20260923 19:30:12.676040 8143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9ecbfef70f47aa96698f087a7a65ca" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aabbf67c1df49269bea3b750e81ebe8"
I20260923 19:30:12.676229 8143 raft_consensus.cc:3037] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.676185 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9ecbfef70f47aa96698f087a7a65ca" candidate_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02"
I20260923 19:30:12.676441 8427 raft_consensus.cc:3037] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.678685 8143 raft_consensus.cc:2454] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 1.
I20260923 19:30:12.679071 8213 leader_election.cc:302] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56662f8ce94c4dc49bbc8c4a1cb74b31, 6aabbf67c1df49269bea3b750e81ebe8; no voters:
I20260923 19:30:12.679376 8565 raft_consensus.cc:2781] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:12.679539 8565 raft_consensus.cc:686] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 LEADER]: Becoming Leader. State: Replica: 56662f8ce94c4dc49bbc8c4a1cb74b31, State: Running, Role: LEADER
I20260923 19:30:12.679783 8427 raft_consensus.cc:2454] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56662f8ce94c4dc49bbc8c4a1cb74b31 in term 1.
I20260923 19:30:12.679903 8565 consensus_queue.cc:237] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.682505 7984 catalog_manager.cc:5495] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } health_report { overall_health: UNKNOWN } } }
I20260923 19:30:12.696091 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.691557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.0.97.2:41091)
0923 19:30:12.691757 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.691763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.691781 (+ 18us) server_negotiation.cc:407] Connection header received
0923 19:30:12.691849 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.691855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.691925 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.691998 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:12.692341 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.693093 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.694407 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.694837 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.694944 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.695327 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.695431 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.695503 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.695668 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.695772 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 19:30:12.699184 8143 raft_consensus.cc:1261] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Refusing update from remote peer 56662f8ce94c4dc49bbc8c4a1cb74b31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:12.699329 8427 raft_consensus.cc:1261] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Refusing update from remote peer 56662f8ce94c4dc49bbc8c4a1cb74b31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:12.699916 8565 consensus_queue.cc:1035] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:12.700498 8565 consensus_queue.cc:1035] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:12.711275 7984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:12.711501 7984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:12.716717 7984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a: Sending DeleteTablet for 3 replicas of tablet af9ecbfef70f47aa96698f087a7a65ca
I20260923 19:30:12.718500 8401 tablet_service.cc:1514] Processing DeleteTablet for tablet af9ecbfef70f47aa96698f087a7a65ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
I20260923 19:30:12.719300 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet af9ecbfef70f47aa96698f087a7a65ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
I20260923 19:30:12.719336 8263 tablet_service.cc:1514] Processing DeleteTablet for tablet af9ecbfef70f47aa96698f087a7a65ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59138
I20260923 19:30:12.719583 8262 tablet_service.cc:1514] Processing DeleteTablet for tablet af9ecbfef70f47aa96698f087a7a65ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59138
I20260923 19:30:12.719777 8567 tablet_replica.cc:330] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: stopping tablet replica
I20260923 19:30:12.719962 8567 raft_consensus.cc:2229] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:30:12.720299 8567 raft_consensus.cc:2258] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:12.720543 7986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 19:30:12.720959 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet af9ecbfef70f47aa96698f087a7a65ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53472
W20260923 19:30:12.721159 7986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:30:12.721222 8566 tablet_replica.cc:330] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: stopping tablet replica
I20260923 19:30:12.721400 8566 raft_consensus.cc:2229] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:12.721585 8566 raft_consensus.cc:2258] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:12.722826 8568 tablet_replica.cc:330] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: stopping tablet replica
I20260923 19:30:12.722865 8566 ts_tablet_manager.cc:1905] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:12.723129 8568 raft_consensus.cc:2229] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:12.723388 8568 raft_consensus.cc:2258] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:12.725188 8568 ts_tablet_manager.cc:1905] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:12.727540 8566 ts_tablet_manager.cc:1918] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:12.727645 8566 log.cc:1196] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/af9ecbfef70f47aa96698f087a7a65ca
I20260923 19:30:12.727896 8566 ts_tablet_manager.cc:1939] T af9ecbfef70f47aa96698f087a7a65ca P 6aabbf67c1df49269bea3b750e81ebe8: Deleting consensus metadata
I20260923 19:30:12.728854 8567 ts_tablet_manager.cc:1905] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:12.731864 7970 catalog_manager.cc:4841] TS 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179): tablet af9ecbfef70f47aa96698f087a7a65ca (table test-table [id=2c16a8d00a9644e39067ef665c4b010d]) successfully deleted
I20260923 19:30:12.735796 8403 tablet_service.cc:1467] Processing CreateTablet for tablet c840acf5736141a3b31da0a5be99bb4a (DEFAULT_TABLE table=test-table [id=73f2cdadacea4c5a81c4ce987d5974fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:12.736303 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c840acf5736141a3b31da0a5be99bb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:12.737489 8262 tablet_service.cc:1467] Processing CreateTablet for tablet c840acf5736141a3b31da0a5be99bb4a (DEFAULT_TABLE table=test-table [id=73f2cdadacea4c5a81c4ce987d5974fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:12.738025 8122 tablet_service.cc:1467] Processing CreateTablet for tablet c840acf5736141a3b31da0a5be99bb4a (DEFAULT_TABLE table=test-table [id=73f2cdadacea4c5a81c4ce987d5974fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:12.738543 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c840acf5736141a3b31da0a5be99bb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:12.744503 8499 tablet_bootstrap.cc:492] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: Bootstrap starting.
I20260923 19:30:12.746230 8499 tablet_bootstrap.cc:654] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:12.746260 8568 ts_tablet_manager.cc:1918] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:12.746469 8568 log.cc:1196] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/af9ecbfef70f47aa96698f087a7a65ca
I20260923 19:30:12.746959 8568 ts_tablet_manager.cc:1939] T af9ecbfef70f47aa96698f087a7a65ca P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting consensus metadata
I20260923 19:30:12.748310 7972 catalog_manager.cc:4841] TS c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099): tablet af9ecbfef70f47aa96698f087a7a65ca (table test-table [id=2c16a8d00a9644e39067ef665c4b010d]) successfully deleted
I20260923 19:30:12.749562 8262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c840acf5736141a3b31da0a5be99bb4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:12.749717 8498 tablet_bootstrap.cc:492] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: Bootstrap starting.
I20260923 19:30:12.751704 8498 tablet_bootstrap.cc:654] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:12.756392 8496 tablet_bootstrap.cc:492] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: Bootstrap starting.
W20260923 19:30:12.756383 7972 catalog_manager.cc:4808] TS c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099): delete failed for tablet af9ecbfef70f47aa96698f087a7a65ca because the tablet was not found. No further retry: Not found: Tablet not found: af9ecbfef70f47aa96698f087a7a65ca
I20260923 19:30:12.758240 8496 tablet_bootstrap.cc:654] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:12.762594 8498 tablet_bootstrap.cc:492] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: No bootstrap required, opened a new log
I20260923 19:30:12.763135 8498 ts_tablet_manager.cc:1397] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260923 19:30:12.764144 8498 raft_consensus.cc:348] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.764472 8498 raft_consensus.cc:374] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:12.764580 8498 raft_consensus.cc:729] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d4138521c24c8e8b4a9772aa0b6a02, State: Initialized, Role: FOLLOWER
I20260923 19:30:12.764887 8498 consensus_queue.cc:260] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.766237 8498 ts_tablet_manager.cc:1428] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 19:30:12.766245 8567 ts_tablet_manager.cc:1918] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:12.767267 8567 log.cc:1196] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/af9ecbfef70f47aa96698f087a7a65ca
I20260923 19:30:12.767778 8567 ts_tablet_manager.cc:1939] T af9ecbfef70f47aa96698f087a7a65ca P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting consensus metadata
I20260923 19:30:12.768810 7970 catalog_manager.cc:4841] TS 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091): tablet af9ecbfef70f47aa96698f087a7a65ca (table test-table [id=2c16a8d00a9644e39067ef665c4b010d]) successfully deleted
W20260923 19:30:12.769160 7970 catalog_manager.cc:4808] TS 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091): delete failed for tablet af9ecbfef70f47aa96698f087a7a65ca because the tablet was not found. No further retry: Not found: Tablet not found: af9ecbfef70f47aa96698f087a7a65ca
I20260923 19:30:12.771907 8496 tablet_bootstrap.cc:492] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: No bootstrap required, opened a new log
I20260923 19:30:12.772059 8496 ts_tablet_manager.cc:1397] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.003s
I20260923 19:30:12.772723 8496 raft_consensus.cc:348] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.772934 8496 raft_consensus.cc:374] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:12.773010 8496 raft_consensus.cc:729] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56662f8ce94c4dc49bbc8c4a1cb74b31, State: Initialized, Role: FOLLOWER
I20260923 19:30:12.773245 8496 consensus_queue.cc:260] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.773727 8496 ts_tablet_manager.cc:1428] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 19:30:12.775055 8499 tablet_bootstrap.cc:492] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: No bootstrap required, opened a new log
I20260923 19:30:12.775233 8499 ts_tablet_manager.cc:1397] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20260923 19:30:12.776038 8499 raft_consensus.cc:348] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.776291 8499 raft_consensus.cc:374] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:12.776373 8499 raft_consensus.cc:729] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aabbf67c1df49269bea3b750e81ebe8, State: Initialized, Role: FOLLOWER
I20260923 19:30:12.776608 8499 consensus_queue.cc:260] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.777165 8499 ts_tablet_manager.cc:1428] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 19:30:12.838383 8504 raft_consensus.cc:482] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:12.838581 8504 raft_consensus.cc:504] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.839253 8504 leader_election.cc:288] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091), c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099)
I20260923 19:30:12.839833 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c840acf5736141a3b31da0a5be99bb4a" candidate_uuid: "6aabbf67c1df49269bea3b750e81ebe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" is_pre_election: true
I20260923 19:30:12.840078 8427 raft_consensus.cc:2454] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aabbf67c1df49269bea3b750e81ebe8 in term 0.
I20260923 19:30:12.839959 8282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c840acf5736141a3b31da0a5be99bb4a" candidate_uuid: "6aabbf67c1df49269bea3b750e81ebe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" is_pre_election: true
I20260923 19:30:12.840303 8282 raft_consensus.cc:2454] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aabbf67c1df49269bea3b750e81ebe8 in term 0.
I20260923 19:30:12.840624 8075 leader_election.cc:302] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aabbf67c1df49269bea3b750e81ebe8, c4d4138521c24c8e8b4a9772aa0b6a02; no voters:
I20260923 19:30:12.840942 8504 raft_consensus.cc:2781] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:30:12.841061 8504 raft_consensus.cc:482] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:30:12.841149 8504 raft_consensus.cc:3037] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.843513 8504 raft_consensus.cc:504] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.844120 8504 leader_election.cc:288] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [CANDIDATE]: Term 1 election: Requested vote from peers 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091), c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099)
I20260923 19:30:12.844705 8282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c840acf5736141a3b31da0a5be99bb4a" candidate_uuid: "6aabbf67c1df49269bea3b750e81ebe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31"
I20260923 19:30:12.844722 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c840acf5736141a3b31da0a5be99bb4a" candidate_uuid: "6aabbf67c1df49269bea3b750e81ebe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02"
I20260923 19:30:12.844898 8427 raft_consensus.cc:3037] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.844910 8282 raft_consensus.cc:3037] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:12.846971 8427 raft_consensus.cc:2454] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aabbf67c1df49269bea3b750e81ebe8 in term 1.
I20260923 19:30:12.847193 8282 raft_consensus.cc:2454] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aabbf67c1df49269bea3b750e81ebe8 in term 1.
I20260923 19:30:12.847400 8075 leader_election.cc:302] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aabbf67c1df49269bea3b750e81ebe8, c4d4138521c24c8e8b4a9772aa0b6a02; no voters:
I20260923 19:30:12.847723 8504 raft_consensus.cc:2781] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:12.848194 8504 raft_consensus.cc:686] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 LEADER]: Becoming Leader. State: Replica: 6aabbf67c1df49269bea3b750e81ebe8, State: Running, Role: LEADER
I20260923 19:30:12.848662 8504 consensus_queue.cc:237] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } }
I20260923 19:30:12.852259 7986 catalog_manager.cc:5495] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6aabbf67c1df49269bea3b750e81ebe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabbf67c1df49269bea3b750e81ebe8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 } health_report { overall_health: UNKNOWN } } }
I20260923 19:30:12.891752 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.885684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.97.1:35179)
0923 19:30:12.886338 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.886345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.886366 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:30:12.886469 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.886472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.886563 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.886734 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:12.887054 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.887917 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.889294 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.889754 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.889860 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.890224 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.891109 (+ 885us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.891229 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.891327 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.891409 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":351,"threads_started":1}
I20260923 19:30:12.907424 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.902436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38449 (local address 127.0.97.3:43099)
0923 19:30:12.902693 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.902701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.903193 (+ 492us) server_negotiation.cc:407] Connection header received
0923 19:30:12.903350 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.903354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.903431 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.903535 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:12.903978 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.904657 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.905982 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.906420 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.906515 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.906876 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.906976 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.907040 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.907141 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.907201 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 19:30:12.907557 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.902418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43099 (local address 127.0.97.1:38449)
0923 19:30:12.903083 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:12.903102 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:12.903236 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:12.903565 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:12.903568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:12.903578 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:30:12.903860 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.903868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.904761 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.904764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.905859 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.905870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.906557 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.906562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.906679 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.906737 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.907052 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.907059 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 19:30:12.907116 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 19:30:12.907221 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":337,"threads_started":1}
I20260923 19:30:12.916294 8282 raft_consensus.cc:1261] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Refusing update from remote peer 6aabbf67c1df49269bea3b750e81ebe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:12.916296 8427 raft_consensus.cc:1261] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Refusing update from remote peer 6aabbf67c1df49269bea3b750e81ebe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:12.916989 8591 consensus_queue.cc:1035] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56662f8ce94c4dc49bbc8c4a1cb74b31" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:12.917346 8504 consensus_queue.cc:1035] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d4138521c24c8e8b4a9772aa0b6a02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:12.942257 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.937024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33083 (local address 127.0.97.1:35179)
0923 19:30:12.937415 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.937420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:12.937433 (+ 13us) server_negotiation.cc:407] Connection header received
0923 19:30:12.937491 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:12.937496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:12.937588 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:12.937678 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:12.938077 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.938939 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.940409 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.940907 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.941035 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.941599 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.941744 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.941830 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:12.941944 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 19:30:12.942027 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260923 19:30:12.942361 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.936943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35179 (local address 127.0.97.3:33083)
0923 19:30:12.937131 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:12.937152 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:12.937246 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:12.937733 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:12.937737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:12.937753 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 19:30:12.937958 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.937966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.939079 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.939083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.940258 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:12.940273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:12.941141 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:12.941145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:12.941352 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:12.941411 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 19:30:12.941837 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 19:30:12.941845 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 19:30:12.941906 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 19:30:12.942055 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 19:30:12.977454 7985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:12.977746 7985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:12.981302 7985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64ff20b6bb7745c0854376598e49013a: Sending DeleteTablet for 3 replicas of tablet c840acf5736141a3b31da0a5be99bb4a
I20260923 19:30:12.982353 8262 tablet_service.cc:1514] Processing DeleteTablet for tablet c840acf5736141a3b31da0a5be99bb4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59138
I20260923 19:30:12.982771 8567 tablet_replica.cc:330] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: stopping tablet replica
I20260923 19:30:12.983022 8567 raft_consensus.cc:2229] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:12.983070 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet c840acf5736141a3b31da0a5be99bb4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53472
I20260923 19:30:12.983291 8567 raft_consensus.cc:2258] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:12.983369 8566 tablet_replica.cc:330] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: stopping tablet replica
I20260923 19:30:12.983544 8566 raft_consensus.cc:2229] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:30:12.983913 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet c840acf5736141a3b31da0a5be99bb4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
I20260923 19:30:12.984091 8566 raft_consensus.cc:2258] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:12.984254 8568 tablet_replica.cc:330] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: stopping tablet replica
I20260923 19:30:12.984440 8568 raft_consensus.cc:2229] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:12.984699 8568 raft_consensus.cc:2258] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:12.985728 8566 ts_tablet_manager.cc:1905] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:12.986012 8567 ts_tablet_manager.cc:1905] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:12.989665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 8057
I20260923 19:30:12.993431 8566 ts_tablet_manager.cc:1918] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:30:12.994089 8567 ts_tablet_manager.cc:1918] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:30:12.994261 8567 log.cc:1196] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/c840acf5736141a3b31da0a5be99bb4a
I20260923 19:30:12.994589 8568 ts_tablet_manager.cc:1905] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:12.994706 8567 ts_tablet_manager.cc:1939] T c840acf5736141a3b31da0a5be99bb4a P 56662f8ce94c4dc49bbc8c4a1cb74b31: Deleting consensus metadata
I20260923 19:30:12.996116 7970 catalog_manager.cc:4841] TS 56662f8ce94c4dc49bbc8c4a1cb74b31 (127.0.97.2:41091): tablet c840acf5736141a3b31da0a5be99bb4a (table test-table [id=73f2cdadacea4c5a81c4ce987d5974fa]) successfully deleted
I20260923 19:30:12.998744 8566 log.cc:1196] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/c840acf5736141a3b31da0a5be99bb4a
I20260923 19:30:12.999284 8566 ts_tablet_manager.cc:1939] T c840acf5736141a3b31da0a5be99bb4a P 6aabbf67c1df49269bea3b750e81ebe8: Deleting consensus metadata
I20260923 19:30:13.000829 7970 catalog_manager.cc:4841] TS 6aabbf67c1df49269bea3b750e81ebe8 (127.0.97.1:35179): tablet c840acf5736141a3b31da0a5be99bb4a (table test-table [id=73f2cdadacea4c5a81c4ce987d5974fa]) successfully deleted
I20260923 19:30:13.002085 8568 ts_tablet_manager.cc:1918] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 19:30:13.002278 8568 log.cc:1196] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/c840acf5736141a3b31da0a5be99bb4a
I20260923 19:30:13.002763 8568 ts_tablet_manager.cc:1939] T c840acf5736141a3b31da0a5be99bb4a P c4d4138521c24c8e8b4a9772aa0b6a02: Deleting consensus metadata
I20260923 19:30:13.003912 7972 catalog_manager.cc:4841] TS c4d4138521c24c8e8b4a9772aa0b6a02 (127.0.97.3:43099): tablet c840acf5736141a3b31da0a5be99bb4a (table test-table [id=73f2cdadacea4c5a81c4ce987d5974fa]) successfully deleted
Sep 23 19:30:13 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260923 19:30:13.021889 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:12.992119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.0.97.1:35179)
0923 19:30:12.992373 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:12.992379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:13.001122 (+ 8743us) server_negotiation.cc:407] Connection header received
0923 19:30:13.001437 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:13.001443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:13.001583 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:13.001675 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:13.003728 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:13.004947 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:13.006928 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:13.007756 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:13.007915 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:13.013735 (+ 5820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:13.013763 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:13.013790 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:13.013851 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:13.018331 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:13.019307 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:13.019320 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:13.019325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:13.019536 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:13.019950 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:13.019955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:13.019958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:13.020526 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:13.020878 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:13.021175 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 19:30:13.021435 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 19:30:13.023026 8188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:13.132215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 8057
I20260923 19:30:13.149554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 8197
Sep 23 19:30:13 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260923 19:30:13.165511 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:13.150470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.0.97.2:41091)
0923 19:30:13.150680 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:13.150685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:13.150706 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:30:13.150935 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:13.150941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:13.151026 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:13.151135 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:13.152392 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:13.153305 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:13.154682 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:13.155118 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:13.155218 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:13.159483 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:13.159500 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:13.159520 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:13.159572 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:13.162896 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:13.163586 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:13.163594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:13.163597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:13.163736 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:13.164093 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:13.164096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:13.164098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:13.164497 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:13.164743 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:13.165017 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 19:30:13.165192 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 19:30:13.166311 8327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:13.304661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 8197
I20260923 19:30:13.323406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 8337
Sep 23 19:30:13 dist-test-slave-kxfb krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260923 19:30:13.338485 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:13.324250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.0.97.3:43099)
0923 19:30:13.324478 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:13.324483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:13.324556 (+ 73us) server_negotiation.cc:407] Connection header received
0923 19:30:13.324759 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:13.324763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:13.324839 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:13.324967 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:13.326169 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:13.326966 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:13.328219 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:13.328629 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:13.328741 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:13.333016 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:13.333032 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:13.333052 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:13.333089 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:13.336001 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:13.336608 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:13.336615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:13.336618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:13.336742 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:13.337027 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:13.337030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:13.337032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:13.337405 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:13.337676 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:13.337909 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 19:30:13.338085 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 19:30:13.339238 8467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:13.461062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 8337
I20260923 19:30:13.481427 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 7954
I20260923 19:30:13.482805 8026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:13.611191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 7954
2026-09-23T19:30:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5653 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:30:13 dist-test-slave-kxfb krb5kdc[8609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:30:13 dist-test-slave-kxfb krb5kdc[8609](info): set up 2 sockets
Sep 23 19:30:13 dist-test-slave-kxfb krb5kdc[8609](info): commencing operation
krb5kdc: starting...
W20260923 19:30:14.631429 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:30:14 dist-test-slave-kxfb krb5kdc[8609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:30:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:30:14.874131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33105
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:15.234566 8622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:15.234946 8622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:15.235097 8622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:15.243556 8622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:30:15.243688 8622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:15.243736 8622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:15.243816 8622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:30:15.243885 8622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:30:15.253983 8622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:15.255344 8622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:15.256697 8622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:15.269721 8628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:15.269831 8627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:15.269831 8630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:15.271023 8622 server_base.cc:1034] running on GCE node
I20260923 19:30:15.272017 8622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:15.273818 8622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:15.275049 8622 hybrid_clock.cc:648] HybridClock initialized: now 1790191815274984 us; error 61 us; skew 500 ppm
I20260923 19:30:15.275461 8622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T19:30:15Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1736 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:30:15 dist-test-slave-kxfb krb5kdc[8636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:30:15 dist-test-slave-kxfb krb5kdc[8636](info): set up 2 sockets
Sep 23 19:30:15 dist-test-slave-kxfb krb5kdc[8636](info): commencing operation
krb5kdc: starting...
W20260923 19:30:16.375401 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:30:16 dist-test-slave-kxfb krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:30:16.622359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46821
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:16.987917 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:16.988269 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:16.988408 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:16.996142 8649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:30:16.996265 8649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:16.996328 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:16.996428 8649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:30:16.996488 8649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:30:17.007480 8649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:17.009138 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:17.010756 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:17.023540 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:17.023569 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:17.023510 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:17.024345 8649 server_base.cc:1034] running on GCE node
I20260923 19:30:17.025267 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:17.026999 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:17.028204 8649 hybrid_clock.cc:648] HybridClock initialized: now 1790191817028150 us; error 48 us; skew 500 ppm
I20260923 19:30:17.028558 8649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:17 dist-test-slave-kxfb krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:17.031980 8649 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:30:17.033710 8649 webserver.cc:460] Webserver started at http://127.0.97.62:41411/ using document root <none> and password file <none>
I20260923 19:30:17.034619 8649 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:17.034766 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:17.035125 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:17.038756 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "043042f8971b408f9feeb39f90f8ae65"
format_stamp: "Formatted at 2026-09-23 19:30:17 on dist-test-slave-kxfb"
server_key: "bf47b058dc72319275877b0517b27796"
server_key_iv: "e61344481a340502c91f2fe0122ead91"
server_key_version: "encryptionkey@0"
I20260923 19:30:17.039665 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "043042f8971b408f9feeb39f90f8ae65"
format_stamp: "Formatted at 2026-09-23 19:30:17 on dist-test-slave-kxfb"
server_key: "bf47b058dc72319275877b0517b27796"
server_key_iv: "e61344481a340502c91f2fe0122ead91"
server_key_version: "encryptionkey@0"
I20260923 19:30:17.045719 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 19:30:17.049805 8664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:17.051491 8649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:30:17.051680 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "043042f8971b408f9feeb39f90f8ae65"
format_stamp: "Formatted at 2026-09-23 19:30:17 on dist-test-slave-kxfb"
server_key: "bf47b058dc72319275877b0517b27796"
server_key_iv: "e61344481a340502c91f2fe0122ead91"
server_key_version: "encryptionkey@0"
I20260923 19:30:17.051891 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:17.073653 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:17.078801 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:17.079119 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:17.098819 8649 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46821
I20260923 19:30:17.098845 8715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46821 every 8 connection(s)
I20260923 19:30:17.100435 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:30:17.104725 8716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:17.107861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 8649
I20260923 19:30:17.108088 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:30:17.108724 388 external_mini_cluster.cc:1428] Setting key 956d9a72f6581bb85fad512f3d985dbc
I20260923 19:30:17.118321 8716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: Bootstrap starting.
I20260923 19:30:17.122325 8716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:17.123474 8716 log.cc:824] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:17.126598 8716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: No bootstrap required, opened a new log
Sep 23 19:30:17 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:30:17.137265 8716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } }
I20260923 19:30:17.137784 8716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:17.137878 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:17.110857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.0.97.62:46821)
0923 19:30:17.111648 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:17.111664 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:17.111718 (+ 54us) server_negotiation.cc:407] Connection header received
0923 19:30:17.113855 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:17.113886 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:17.114512 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:17.114999 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:17.116394 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:17.118420 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:17.119702 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:17.120775 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:17.120955 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:17.128690 (+ 7735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:17.128700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:17.128723 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:17.128779 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:17.132942 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:17.133677 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:17.133686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:17.133695 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:17.133906 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:17.134256 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:17.134261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:17.134264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:17.135119 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:17.135435 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:17.135838 (+ 403us) server_negotiation.cc:299] Negotiation successful
0923 19:30:17.136366 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":322,"threads_started":1}
I20260923 19:30:17.137912 8716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043042f8971b408f9feeb39f90f8ae65, State: Initialized, Role: FOLLOWER
I20260923 19:30:17.138615 8716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } }
I20260923 19:30:17.138866 8716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:30:17.139001 8716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:30:17.139181 8716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:17.141602 8716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } }
I20260923 19:30:17.142148 8716 leader_election.cc:302] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043042f8971b408f9feeb39f90f8ae65; no voters:
I20260923 19:30:17.142802 8716 leader_election.cc:288] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:30:17.142948 8721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:17.143621 8721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 1 LEADER]: Becoming Leader. State: Replica: 043042f8971b408f9feeb39f90f8ae65, State: Running, Role: LEADER
I20260923 19:30:17.144157 8721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } }
I20260923 19:30:17.144927 8716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:30:17.147148 8722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "043042f8971b408f9feeb39f90f8ae65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } } }
I20260923 19:30:17.147112 8723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043042f8971b408f9feeb39f90f8ae65. Latest consensus state: current_term: 1 leader_uuid: "043042f8971b408f9feeb39f90f8ae65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } } }
I20260923 19:30:17.147557 8723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:17.147557 8722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:17.150938 8727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:30:17.156579 8727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:30:17.167307 8727 catalog_manager.cc:1347] Generated new cluster ID: 896c736f8458431c9a75447645fc2738
I20260923 19:30:17.167457 8727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:30:17.186978 8727 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:30:17.188552 8727 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:30:17.204485 8727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: Generated new TSK 0
I20260923 19:30:17.205667 8727 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:30:17.280232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46821
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:17.647637 8742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:17.648023 8742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:17.648161 8742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:17.656139 8742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:17.656288 8742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:17.656469 8742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:30:17.667325 8742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:17.668936 8742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:17.670480 8742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:17.684947 8747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:17.684976 8748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:17.685235 8750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:17.685770 8742 server_base.cc:1034] running on GCE node
I20260923 19:30:17.686535 8742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:17.687628 8742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:17.688836 8742 hybrid_clock.cc:648] HybridClock initialized: now 1790191817688778 us; error 49 us; skew 500 ppm
I20260923 19:30:17.689185 8742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:17 dist-test-slave-kxfb krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:17.692560 8742 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:30:17.694191 8742 webserver.cc:460] Webserver started at http://127.0.97.1:44857/ using document root <none> and password file <none>
I20260923 19:30:17.695026 8742 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:17.695138 8742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:17.695452 8742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:17.698737 8742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "c224c49db11b419f8e9a5d0d584f29c8"
format_stamp: "Formatted at 2026-09-23 19:30:17 on dist-test-slave-kxfb"
server_key: "0a98224f695501f909d20a10f24617e3"
server_key_iv: "a7df2c18ffc502d81eaab28fc4bfe5a8"
server_key_version: "encryptionkey@0"
I20260923 19:30:17.699520 8742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c224c49db11b419f8e9a5d0d584f29c8"
format_stamp: "Formatted at 2026-09-23 19:30:17 on dist-test-slave-kxfb"
server_key: "0a98224f695501f909d20a10f24617e3"
server_key_iv: "a7df2c18ffc502d81eaab28fc4bfe5a8"
server_key_version: "encryptionkey@0"
I20260923 19:30:17.705684 8742 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 19:30:17.710052 8757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:17.711968 8742 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260923 19:30:17.712169 8742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "c224c49db11b419f8e9a5d0d584f29c8"
format_stamp: "Formatted at 2026-09-23 19:30:17 on dist-test-slave-kxfb"
server_key: "0a98224f695501f909d20a10f24617e3"
server_key_iv: "a7df2c18ffc502d81eaab28fc4bfe5a8"
server_key_version: "encryptionkey@0"
I20260923 19:30:17.712406 8742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:17.755383 8742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:17.760205 8742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:17.760499 8742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:17.761552 8742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:17.763113 8742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:17.763190 8742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:17.763288 8742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:17.763335 8742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:17.796059 8742 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33039
I20260923 19:30:17.796144 8869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33039 every 8 connection(s)
I20260923 19:30:17.797530 8742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:30:17.807123 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 8742
I20260923 19:30:17.807348 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:30:17.807989 388 external_mini_cluster.cc:1428] Setting key 20b20865437f2bd323f8203ad86c3dc9
Sep 23 19:30:17 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:30:17.820613 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:17.800607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60943 (local address 127.0.97.62:46821)
0923 19:30:17.801250 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:17.801256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:17.802344 (+ 1088us) server_negotiation.cc:407] Connection header received
0923 19:30:17.804378 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:17.804385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:17.804476 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:17.804589 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:17.806245 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:17.807247 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:17.809277 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:17.809858 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:17.809987 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:17.814362 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:17.814368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:17.814372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:17.814412 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:17.817050 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:17.817977 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:17.817982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:17.817984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:17.818107 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:17.818596 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:17.818601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:17.818603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:17.818945 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:17.819118 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:17.819967 (+ 849us) server_negotiation.cc:299] Negotiation successful
0923 19:30:17.820213 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":325,"threads_started":1}
I20260923 19:30:17.821316 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:17.801249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:60943)
0923 19:30:17.802122 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:17.802172 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:17.804072 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:17.804865 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:17.804874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:17.805434 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:17.806066 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:17.806086 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:17.807493 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:17.807499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:17.809131 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:17.809151 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:17.810256 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:17.810264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:17.810554 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:17.811304 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:17.811328 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:17.814049 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:17.817195 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:17.817205 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:17.817224 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:17.817844 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:17.818218 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:17.818222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:17.818226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:17.818483 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:17.819097 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:17.819105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:17.819520 (+ 415us) client_negotiation.cc:769] Sending connection context
0923 19:30:17.819796 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 19:30:17.820203 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":351,"threads_started":1}
I20260923 19:30:17.822894 8870 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46821
I20260923 19:30:17.823194 8870 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:17.823894 8870 heartbeater.cc:511] Master 127.0.97.62:46821 requested a full tablet report, sending...
I20260923 19:30:17.825942 8681 ts_manager.cc:194] Registered new tserver with Master: c224c49db11b419f8e9a5d0d584f29c8 (127.0.97.1:33039)
I20260923 19:30:17.827468 8681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60943
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:30:17.862637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46821
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:18.230917 8876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:18.231246 8876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:18.231346 8876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:18.239102 8876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:18.239248 8876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:18.239374 8876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:30:18.252177 8876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:18.253626 8876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:18.255023 8876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:18.268795 8881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:18.268735 8884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:18.268867 8882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:18.269881 8876 server_base.cc:1034] running on GCE node
I20260923 19:30:18.270697 8876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:18.271854 8876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:18.273098 8876 hybrid_clock.cc:648] HybridClock initialized: now 1790191818273042 us; error 44 us; skew 500 ppm
I20260923 19:30:18.273458 8876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:18 dist-test-slave-kxfb krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:18.276960 8876 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:30:18.278568 8876 webserver.cc:460] Webserver started at http://127.0.97.2:44317/ using document root <none> and password file <none>
I20260923 19:30:18.279336 8876 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:18.279436 8876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:18.279762 8876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:18.283142 8876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "f22adc69fe104b18a367119a7ba98d6c"
format_stamp: "Formatted at 2026-09-23 19:30:18 on dist-test-slave-kxfb"
server_key: "1ca89a46ae3584a351275d3ecf23e833"
server_key_iv: "79190fd1b888cdc03ab56ad21f7567b7"
server_key_version: "encryptionkey@0"
I20260923 19:30:18.283973 8876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f22adc69fe104b18a367119a7ba98d6c"
format_stamp: "Formatted at 2026-09-23 19:30:18 on dist-test-slave-kxfb"
server_key: "1ca89a46ae3584a351275d3ecf23e833"
server_key_iv: "79190fd1b888cdc03ab56ad21f7567b7"
server_key_version: "encryptionkey@0"
I20260923 19:30:18.289958 8876 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20260923 19:30:18.294052 8891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:18.295831 8876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260923 19:30:18.296052 8876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "f22adc69fe104b18a367119a7ba98d6c"
format_stamp: "Formatted at 2026-09-23 19:30:18 on dist-test-slave-kxfb"
server_key: "1ca89a46ae3584a351275d3ecf23e833"
server_key_iv: "79190fd1b888cdc03ab56ad21f7567b7"
server_key_version: "encryptionkey@0"
I20260923 19:30:18.296270 8876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:18.314958 8876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:18.319734 8876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:18.320045 8876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:18.321097 8876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:18.322796 8876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:18.322891 8876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:18.323040 8876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:18.323102 8876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:18.356913 8876 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33575
I20260923 19:30:18.356945 9003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33575 every 8 connection(s)
I20260923 19:30:18.358423 8876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:30:18.367197 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 8876
I20260923 19:30:18.367383 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:30:18.367839 388 external_mini_cluster.cc:1428] Setting key 3682b06c841fae897b0d7714e509c219
Sep 23 19:30:18 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:30:18.380541 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:18.361201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58525 (local address 127.0.97.62:46821)
0923 19:30:18.361733 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:18.361739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:18.362741 (+ 1002us) server_negotiation.cc:407] Connection header received
0923 19:30:18.364820 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:18.364826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:18.364930 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:18.365033 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:18.366776 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:18.367630 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:18.369449 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:18.370044 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:18.370204 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:18.374068 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:18.374074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:18.374087 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:18.374131 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:18.377218 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:18.378015 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:18.378021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:18.378024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:18.378191 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:18.378690 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:18.378694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:18.378697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:18.379046 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:18.379208 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:18.379934 (+ 726us) server_negotiation.cc:299] Negotiation successful
0923 19:30:18.380169 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":247,"threads_started":1}
I20260923 19:30:18.381659 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:18.361698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:58525)
0923 19:30:18.362555 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:18.362606 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:18.364490 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:18.365314 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:18.365324 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:18.365901 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:18.366611 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:18.366631 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:18.367780 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:18.367784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:18.369284 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:18.369298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:18.370336 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:18.370346 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:18.370603 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:18.371323 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:18.371346 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:18.373809 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:18.377374 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:18.377382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:18.377397 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:18.377862 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:18.378362 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:18.378366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:18.378369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:18.378562 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:18.379165 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:18.379171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:18.379551 (+ 380us) client_negotiation.cc:769] Sending connection context
0923 19:30:18.379801 (+ 250us) client_negotiation.cc:240] Negotiation successful
0923 19:30:18.380278 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":303,"threads_started":1}
I20260923 19:30:18.383440 9004 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46821
I20260923 19:30:18.383771 9004 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:18.384624 9004 heartbeater.cc:511] Master 127.0.97.62:46821 requested a full tablet report, sending...
I20260923 19:30:18.386468 8681 ts_manager.cc:194] Registered new tserver with Master: f22adc69fe104b18a367119a7ba98d6c (127.0.97.2:33575)
I20260923 19:30:18.387642 8681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58525
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:30:18.417546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46821
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:18.787406 9010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:18.787760 9010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:18.787887 9010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:18.795708 9010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:18.795850 9010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:18.796028 9010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:30:18.806759 9010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:18.808341 9010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:18.809842 9010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:18.824185 9016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:18.824215 9018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:18.824124 9015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:18.825439 9010 server_base.cc:1034] running on GCE node
I20260923 19:30:18.826188 9010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:18.827389 9010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:18.828616 9010 hybrid_clock.cc:648] HybridClock initialized: now 1790191818828552 us; error 64 us; skew 500 ppm
I20260923 19:30:18.828962 9010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:30:18.830262 8870 heartbeater.cc:503] Master 127.0.97.62:46821 was elected leader, sending a full tablet report...
Sep 23 19:30:18 dist-test-slave-kxfb krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:18.832760 9010 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:30:18.834573 9010 webserver.cc:460] Webserver started at http://127.0.97.3:43191/ using document root <none> and password file <none>
I20260923 19:30:18.835376 9010 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:18.835498 9010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:18.835821 9010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:18.839234 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "5036ab1c775b43a4be051b7d291524da"
format_stamp: "Formatted at 2026-09-23 19:30:18 on dist-test-slave-kxfb"
server_key: "bfa26c5080434f551a31f8be37d80c91"
server_key_iv: "d2997cf5202935426dd7c7375b435e89"
server_key_version: "encryptionkey@0"
I20260923 19:30:18.840102 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5036ab1c775b43a4be051b7d291524da"
format_stamp: "Formatted at 2026-09-23 19:30:18 on dist-test-slave-kxfb"
server_key: "bfa26c5080434f551a31f8be37d80c91"
server_key_iv: "d2997cf5202935426dd7c7375b435e89"
server_key_version: "encryptionkey@0"
I20260923 19:30:18.846196 9010 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 19:30:18.850179 9025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:18.852059 9010 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260923 19:30:18.852249 9010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "5036ab1c775b43a4be051b7d291524da"
format_stamp: "Formatted at 2026-09-23 19:30:18 on dist-test-slave-kxfb"
server_key: "bfa26c5080434f551a31f8be37d80c91"
server_key_iv: "d2997cf5202935426dd7c7375b435e89"
server_key_version: "encryptionkey@0"
I20260923 19:30:18.852475 9010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:18.894024 9010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:18.899188 9010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:18.899546 9010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:18.900807 9010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:18.902586 9010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:18.902678 9010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:18.902845 9010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:18.902971 9010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:18.938386 9010 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44805
I20260923 19:30:18.938476 9137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44805 every 8 connection(s)
I20260923 19:30:18.939870 9010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:30:18.946053 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 9010
I20260923 19:30:18.946270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:30:18.946828 388 external_mini_cluster.cc:1428] Setting key 9588467aaa69657f301bd2941df226bb
Sep 23 19:30:18 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:30:18.962919 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:18.943107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49229 (local address 127.0.97.62:46821)
0923 19:30:18.943668 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:18.943673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:18.944812 (+ 1139us) server_negotiation.cc:407] Connection header received
0923 19:30:18.947066 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:18.947075 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:18.947209 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:18.947382 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:18.949277 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:18.950423 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:18.952238 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:18.952747 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:18.952858 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:18.956418 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:18.956425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:18.956429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:18.956472 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:18.959403 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:18.960164 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:18.960168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:18.960171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:18.960309 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:18.960718 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:18.960723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:18.960727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:18.961134 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:18.961274 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:18.962184 (+ 910us) server_negotiation.cc:299] Negotiation successful
0923 19:30:18.962514 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":265,"threads_started":1}
I20260923 19:30:18.963876 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:18.943696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:49229)
0923 19:30:18.944543 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:18.944680 (+ 137us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:18.946673 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:18.947768 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:18.947778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:18.948328 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:18.949086 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:18.949109 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:18.950633 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:18.950646 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:18.952081 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:18.952110 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:18.952987 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:18.952992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:18.953208 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:18.953942 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:18.953967 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:18.956155 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:18.959567 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:18.959574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:18.959586 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:18.960042 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:18.960399 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:18.960402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:18.960404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:18.960578 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:18.961303 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:18.961312 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:18.961720 (+ 408us) client_negotiation.cc:769] Sending connection context
0923 19:30:18.962035 (+ 315us) client_negotiation.cc:240] Negotiation successful
0923 19:30:18.962568 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":324,"threads_started":1}
I20260923 19:30:18.965452 9138 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46821
I20260923 19:30:18.965873 9138 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:18.966900 9138 heartbeater.cc:511] Master 127.0.97.62:46821 requested a full tablet report, sending...
I20260923 19:30:18.968590 8681 ts_manager.cc:194] Registered new tserver with Master: 5036ab1c775b43a4be051b7d291524da (127.0.97.3:44805)
I20260923 19:30:18.969492 8681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49229
I20260923 19:30:18.976258 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260923 19:30:19.125707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 8742
Sep 23 19:30:19 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:30:19.143452 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:19.127191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.0.97.1:33039)
0923 19:30:19.127948 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:19.127957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:19.127992 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:30:19.128072 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:19.128080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:19.128343 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:19.128524 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:19.129947 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:19.131210 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:19.132544 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:19.133220 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:19.133337 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:19.137414 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:19.137430 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:19.137449 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:19.137509 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:19.140685 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:19.141373 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:19.141381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:19.141383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:19.141553 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:19.141864 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:19.141867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:19.141869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:19.142280 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:19.142625 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:19.142810 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 19:30:19.143048 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":392,"threads_started":1}
I20260923 19:30:19.145043 8865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:19.236833 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 8742
I20260923 19:30:19.252046 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 8876
Sep 23 19:30:19 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:30:19.270031 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:19.253590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.0.97.2:33575)
0923 19:30:19.254356 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:19.254368 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:19.254410 (+ 42us) server_negotiation.cc:407] Connection header received
0923 19:30:19.254499 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:19.254507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:19.254780 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:19.254948 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:19.256197 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:19.257503 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:19.258889 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:19.259529 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:19.259637 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:19.263815 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:19.263831 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:19.263849 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:19.263896 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:19.267092 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:19.267935 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:19.267942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:19.267945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:19.268080 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:19.268415 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:19.268419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:19.268421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:19.268810 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:19.269061 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:19.269518 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 19:30:19.269740 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":357,"threads_started":1}
I20260923 19:30:19.271816 8999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:19.359081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 8876
I20260923 19:30:19.374159 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9010
Sep 23 19:30:19 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:30:19.391884 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:19.375300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.0.97.3:44805)
0923 19:30:19.375856 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:19.375910 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:19.375944 (+ 34us) server_negotiation.cc:407] Connection header received
0923 19:30:19.376022 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:19.376029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:19.376295 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:19.376438 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:19.377812 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:19.379050 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:19.380401 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:19.381118 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:19.381237 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:19.385714 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:19.385727 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:19.385745 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:19.385792 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:19.388767 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:19.389486 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:19.389495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:19.389497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:19.389654 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:19.390069 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:19.390075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:19.390077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:19.390620 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:19.390952 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:19.391214 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 19:30:19.391499 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
I20260923 19:30:19.393334 9133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:19.479871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 9010
I20260923 19:30:19.494027 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 8649
I20260923 19:30:19.495400 8711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:19.606483 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 8649
I20260923 19:30:19.626946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46821
--webserver_interface=127.0.97.62
--webserver_port=41411
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:19.996421 9156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:19.996819 9156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:19.996960 9156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:20.004825 9156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:30:20.005012 9156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:20.005105 9156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:20.005185 9156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:30:20.005247 9156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:30:20.016567 9156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:20.018345 9156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:20.019925 9156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:20.032399 9161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:20.032702 9164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:20.032757 9162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:20.033560 9156 server_base.cc:1034] running on GCE node
I20260923 19:30:20.034595 9156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:20.036339 9156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:20.037580 9156 hybrid_clock.cc:648] HybridClock initialized: now 1790191820037510 us; error 58 us; skew 500 ppm
I20260923 19:30:20.037990 9156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:20 dist-test-slave-kxfb krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:20.041698 9156 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:30:20.043601 9156 webserver.cc:460] Webserver started at http://127.0.97.62:41411/ using document root <none> and password file <none>
I20260923 19:30:20.044566 9156 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:20.044745 9156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:20.051414 9156 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 19:30:20.054792 9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:20.056566 9156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 19:30:20.056764 9156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "043042f8971b408f9feeb39f90f8ae65"
format_stamp: "Formatted at 2026-09-23 19:30:17 on dist-test-slave-kxfb"
server_key: "bf47b058dc72319275877b0517b27796"
server_key_iv: "e61344481a340502c91f2fe0122ead91"
server_key_version: "encryptionkey@0"
I20260923 19:30:20.057910 9156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:20.082038 9156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:20.086691 9156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:20.086973 9156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:20.106748 9156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46821
I20260923 19:30:20.106772 9222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46821 every 8 connection(s)
I20260923 19:30:20.108539 9156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:30:20.113701 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 9156
I20260923 19:30:20.114579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33039
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46821
--webserver_port=44857
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:30:20.116423 9223 sys_catalog.cc:263] Verifying existing consensus state
I20260923 19:30:20.120059 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: Bootstrap starting.
I20260923 19:30:20.141666 9223 log.cc:824] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:20.152024 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 19:30:20.152612 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: Bootstrap complete.
I20260923 19:30:20.162366 9223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } }
I20260923 19:30:20.163414 9223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043042f8971b408f9feeb39f90f8ae65, State: Initialized, Role: FOLLOWER
I20260923 19:30:20.164139 9223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } }
I20260923 19:30:20.164382 9223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:30:20.164500 9223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:30:20.164677 9223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 1 FOLLOWER]: Advancing to term 2
I20260923 19:30:20.169059 9223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } }
I20260923 19:30:20.169701 9223 leader_election.cc:302] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043042f8971b408f9feeb39f90f8ae65; no voters:
I20260923 19:30:20.170432 9223 leader_election.cc:288] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 19:30:20.170609 9227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 19:30:20.171638 9227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [term 2 LEADER]: Becoming Leader. State: Replica: 043042f8971b408f9feeb39f90f8ae65, State: Running, Role: LEADER
I20260923 19:30:20.172242 9227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } }
I20260923 19:30:20.172871 9223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:30:20.176040 9229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043042f8971b408f9feeb39f90f8ae65. Latest consensus state: current_term: 2 leader_uuid: "043042f8971b408f9feeb39f90f8ae65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } } }
I20260923 19:30:20.176566 9229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:20.176036 9228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "043042f8971b408f9feeb39f90f8ae65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043042f8971b408f9feeb39f90f8ae65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46821 } } }
I20260923 19:30:20.176730 9228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:20.184183 9234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:30:20.191030 9234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:30:20.192816 9234 catalog_manager.cc:1259] Loaded cluster ID: 896c736f8458431c9a75447645fc2738
I20260923 19:30:20.193048 9234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:30:20.197515 9234 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:30:20.200173 9234 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 043042f8971b408f9feeb39f90f8ae65: Loaded TSK: 0
I20260923 19:30:20.202026 9234 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 19:30:20.506016 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:20.506404 9225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:20.506563 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:20.514475 9225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:20.514606 9225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:20.514775 9225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:30:20.525369 9225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44857
--tserver_master_addrs=127.0.97.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:20.527092 9225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:20.528637 9225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:20.543319 9250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:20.543387 9251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:20.543272 9253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:20.544361 9225 server_base.cc:1034] running on GCE node
I20260923 19:30:20.545085 9225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:20.546190 9225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:20.547416 9225 hybrid_clock.cc:648] HybridClock initialized: now 1790191820547316 us; error 86 us; skew 500 ppm
I20260923 19:30:20.547780 9225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:20 dist-test-slave-kxfb krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191820, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:20.551390 9225 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 19:30:20.553133 9225 webserver.cc:460] Webserver started at http://127.0.97.1:44857/ using document root <none> and password file <none>
I20260923 19:30:20.553926 9225 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:20.554069 9225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:20.560235 9225 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:30:20.563475 9260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:20.565094 9225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:30:20.565274 9225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "c224c49db11b419f8e9a5d0d584f29c8"
format_stamp: "Formatted at 2026-09-23 19:30:17 on dist-test-slave-kxfb"
server_key: "0a98224f695501f909d20a10f24617e3"
server_key_iv: "a7df2c18ffc502d81eaab28fc4bfe5a8"
server_key_version: "encryptionkey@0"
I20260923 19:30:20.566375 9225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:20.585722 9225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:20.590693 9225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:20.590998 9225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:20.592154 9225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:20.593626 9225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:20.593721 9225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:20.593796 9225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:20.593859 9225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:20.627079 9225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33039
I20260923 19:30:20.627166 9372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33039 every 8 connection(s)
I20260923 19:30:20.628857 9225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:30:20.630919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 9225
I20260923 19:30:20.631635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33575
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46821
--webserver_port=44317
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:30:20 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:20 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:20.650460 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:20.632703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:46101)
0923 19:30:20.633574 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:20.633616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:20.635438 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:20.636821 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:20.636831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:20.637516 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:20.638325 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:20.638351 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.640582 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.640588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:20.642464 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:20.642480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.643729 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.643738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:20.644052 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:20.644765 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:20.644790 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:20.648839 (+ 4049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":351,"threads_started":1}
W20260923 19:30:20.650447 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:20.632330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46101 (local address 127.0.97.62:46821)
0923 19:30:20.633004 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:20.633016 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:20.633766 (+ 750us) server_negotiation.cc:407] Connection header received
0923 19:30:20.635975 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:20.635986 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:20.636314 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:20.636624 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:20.638637 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.640372 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.642618 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.643399 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.643635 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:20.649266 (+ 5631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46101: BlockingRecv error: recv got EOF from 127.0.97.1:46101 (error 108)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":262,"threads_started":1}
W20260923 19:30:20.651467 9373 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46821 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46821: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 23 19:30:20 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:20 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:20.660215 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:20.652197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:55727)
0923 19:30:20.652359 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:20.652379 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:20.652486 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:20.652891 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:20.652897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:20.653273 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:20.653425 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:20.653434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.654486 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.654491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:20.655427 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:20.655438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.656101 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.656106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:20.656268 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:20.656868 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:20.656885 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:20.659907 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 19:30:20.660393 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:20.652425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55727 (local address 127.0.97.62:46821)
0923 19:30:20.652626 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:20.652630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:20.652645 (+ 15us) server_negotiation.cc:407] Connection header received
0923 19:30:20.652698 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:20.652702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:20.652766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:20.652848 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:20.653561 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.654356 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.655557 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.655963 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.656073 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:20.660183 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55727: BlockingRecv error: recv got EOF from 127.0.97.1:55727 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 19:30:20 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:20 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:20.672808 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:20.661324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:59005)
0923 19:30:20.661486 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:20.661505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:20.661585 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:20.662020 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:20.662025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:20.662533 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:20.662727 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:20.662738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.665099 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.665105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:20.666611 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:20.666626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.667759 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.667766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:20.667962 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:20.668628 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:20.668652 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:20.672467 (+ 3815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 19:30:20.673130 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:20.661587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59005 (local address 127.0.97.62:46821)
0923 19:30:20.661756 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:20.661760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:20.661773 (+ 13us) server_negotiation.cc:407] Connection header received
0923 19:30:20.661815 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:20.661819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:20.661893 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:20.661982 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:20.663844 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.664913 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.666776 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:20.667407 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:20.667538 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:20.672831 (+ 5293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59005: BlockingRecv error: recv got EOF from 127.0.97.1:59005 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 19:30:20.673422 9373 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 19:30:20.986845 9377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:20.987124 9377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:20.987248 9377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:20.994452 9377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:20.994580 9377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:20.994727 9377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:30:21.005174 9377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44317
--tserver_master_addrs=127.0.97.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:21.006577 9377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:21.007891 9377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:21.021986 9382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:21.022080 9385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:21.022063 9383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:21.023077 9377 server_base.cc:1034] running on GCE node
I20260923 19:30:21.023768 9377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:21.024814 9377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:21.026028 9377 hybrid_clock.cc:648] HybridClock initialized: now 1790191821025950 us; error 79 us; skew 500 ppm
I20260923 19:30:21.026394 9377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191821, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:21.029490 9377 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 19:30:21.031031 9377 webserver.cc:460] Webserver started at http://127.0.97.2:44317/ using document root <none> and password file <none>
I20260923 19:30:21.031718 9377 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:21.031829 9377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:21.037459 9377 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:30:21.040536 9392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:21.042182 9377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 19:30:21.042405 9377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "f22adc69fe104b18a367119a7ba98d6c"
format_stamp: "Formatted at 2026-09-23 19:30:18 on dist-test-slave-kxfb"
server_key: "1ca89a46ae3584a351275d3ecf23e833"
server_key_iv: "79190fd1b888cdc03ab56ad21f7567b7"
server_key_version: "encryptionkey@0"
I20260923 19:30:21.043370 9377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:21.073154 9377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:21.077435 9377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:21.077673 9377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:21.078650 9377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:21.079998 9377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:21.080101 9377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:21.080188 9377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:21.080243 9377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:21.114471 9377 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33575
I20260923 19:30:21.114516 9504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33575 every 8 connection(s)
I20260923 19:30:21.115942 9377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:30:21.120536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 9377
I20260923 19:30:21.121428 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44805
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46821
--webserver_port=43191
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 19:30:21.136265 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:21.119340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58909 (local address 127.0.97.62:46821)
0923 19:30:21.119588 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:21.119594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:21.121091 (+ 1497us) server_negotiation.cc:407] Connection header received
0923 19:30:21.124070 (+ 2979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:21.124075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:21.124166 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:21.124280 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:21.126604 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.127352 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.129782 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.130251 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.130411 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.135990 (+ 5579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58909: BlockingRecv error: recv got EOF from 127.0.97.2:58909 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 19:30:21.137135 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:21.119831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:58909)
0923 19:30:21.120834 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:21.120906 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:21.123671 (+ 2765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:21.124630 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:21.124643 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:21.125421 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:21.126375 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.126407 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.127624 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.127629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.129571 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.129604 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.130589 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.130597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.130913 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.131754 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:21.131781 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:21.135706 (+ 3925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":348,"threads_started":1}
W20260923 19:30:21.138166 9505 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46821 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46821: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:21.148363 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:21.139000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:34297)
0923 19:30:21.139212 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:21.139234 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:21.139364 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:21.139771 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:21.139775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:21.140136 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:21.140314 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.140326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.141770 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.141775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.142824 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.142838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.143779 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.143784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.143942 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.144572 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:21.144591 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:21.148046 (+ 3455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260923 19:30:21.148661 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:21.139201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34297 (local address 127.0.97.62:46821)
0923 19:30:21.139437 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:21.139444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:21.139470 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:30:21.139533 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:21.139538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:21.139623 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:21.139748 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:21.140479 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.141603 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.142974 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.143585 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.143749 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.148369 (+ 4620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34297: BlockingRecv error: recv got EOF from 127.0.97.2:34297 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:21.158149 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:21.149540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:48677)
0923 19:30:21.149740 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:21.149764 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:21.149877 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:21.150211 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:21.150215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:21.150569 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:21.150745 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.150754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.152156 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.152161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.153197 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.153210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.154136 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.154141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.154346 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.155020 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:21.155038 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:21.157894 (+ 2856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 19:30:21.158583 9505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 19:30:21.158947 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:21.149699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48677 (local address 127.0.97.62:46821)
0923 19:30:21.149907 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:21.149914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:21.149936 (+ 22us) server_negotiation.cc:407] Connection header received
0923 19:30:21.149994 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:21.149999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:21.150076 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:21.150180 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:21.150881 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.152000 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.153338 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.153977 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.154106 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.158170 (+ 4064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48677: BlockingRecv error: recv got EOF from 127.0.97.2:48677 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 19:30:21.449851 9508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:21.450133 9508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:21.450227 9508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:21.457289 9508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:21.457410 9508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:21.457522 9508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:30:21.467267 9508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43191
--tserver_master_addrs=127.0.97.62:46821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:21.468576 9508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:21.469810 9508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:21.482860 9516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:21.482944 9514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:21.483114 9513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:21.484167 9508 server_base.cc:1034] running on GCE node
I20260923 19:30:21.484853 9508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:21.485890 9508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:21.487087 9508 hybrid_clock.cc:648] HybridClock initialized: now 1790191821487021 us; error 71 us; skew 500 ppm
I20260923 19:30:21.487389 9508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191821, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:21.490530 9508 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 19:30:21.492008 9508 webserver.cc:460] Webserver started at http://127.0.97.3:43191/ using document root <none> and password file <none>
I20260923 19:30:21.492666 9508 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:21.492805 9508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:21.498277 9508 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:30:21.501302 9523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:21.503032 9508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:30:21.503206 9508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "5036ab1c775b43a4be051b7d291524da"
format_stamp: "Formatted at 2026-09-23 19:30:18 on dist-test-slave-kxfb"
server_key: "bfa26c5080434f551a31f8be37d80c91"
server_key_iv: "d2997cf5202935426dd7c7375b435e89"
server_key_version: "encryptionkey@0"
I20260923 19:30:21.504177 9508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:21.523289 9508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:21.527473 9508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:21.527736 9508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:21.528764 9508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:21.530112 9508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:21.530210 9508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:21.530344 9508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:21.530416 9508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:21.564361 9508 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44805
I20260923 19:30:21.564465 9635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44805 every 8 connection(s)
I20260923 19:30:21.565887 9508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:30:21.570479 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 9508
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 19:30:21.583482 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:21.569067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36471 (local address 127.0.97.62:46821)
0923 19:30:21.569354 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:21.569362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:21.571014 (+ 1652us) server_negotiation.cc:407] Connection header received
0923 19:30:21.573336 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:21.573342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:21.573452 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:21.573537 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:21.575351 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.576124 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.577639 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.578238 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.578400 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.583152 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36471: BlockingRecv error: recv got EOF from 127.0.97.3:36471 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"spinlock_wait_cycles":8832}
I20260923 19:30:21.584136 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:21.569704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:36471)
0923 19:30:21.570789 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:21.570852 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:21.573033 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:21.573828 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:21.573841 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:21.574536 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:21.575191 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.575208 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.576273 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.576279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.577502 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.577514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.578488 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.578495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.578749 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.579374 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:21.579404 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:21.582912 (+ 3508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":447,"threads_started":1}
W20260923 19:30:21.584967 9636 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46821 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46821: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260923 19:30:21.589501 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:21.571760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.0.97.62:46821)
0923 19:30:21.573256 (+ 1496us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:21.573261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:21.573279 (+ 18us) server_negotiation.cc:407] Connection header received
0923 19:30:21.573347 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:21.573354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:21.573457 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:21.573568 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:21.575539 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.576478 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.577785 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.578475 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.578612 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.581490 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:21.581501 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:21.581521 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:21.581569 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:21.586038 (+ 4469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:21.586857 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:21.586864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:21.586872 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:21.587062 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:21.587410 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:21.587415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:21.587417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:21.587980 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:21.588288 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:21.588719 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 19:30:21.589125 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1404,"spinlock_wait_cycles":7296,"thread_start_us":1269,"threads_started":1}
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:21.593536 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:21.585655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:49263)
0923 19:30:21.585911 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:21.585930 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:21.586046 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:21.586363 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:21.586368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:21.586720 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:21.586855 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.586863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.587878 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.587883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.588809 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.588821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.589606 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.589612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.589769 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.590271 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:21.590320 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:21.593263 (+ 2943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157}
W20260923 19:30:21.593701 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:21.585783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49263 (local address 127.0.97.62:46821)
0923 19:30:21.585937 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:21.585942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:21.586011 (+ 69us) server_negotiation.cc:407] Connection header received
0923 19:30:21.586155 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:21.586161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:21.586229 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:21.586337 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:21.586990 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.587754 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.588936 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.589471 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.589579 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.593481 (+ 3902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49263: BlockingRecv error: recv got EOF from 127.0.97.3:49263 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:21.601469 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:21.594361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:49089)
0923 19:30:21.594511 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:21.594530 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:21.594616 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:21.595019 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:21.595024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:21.595316 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:21.595451 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.595459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.596381 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.596387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.597269 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.597280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.597919 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.597924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.598079 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.598724 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:21.598742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:21.601192 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 19:30:21.601763 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:21.594482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49089 (local address 127.0.97.62:46821)
0923 19:30:21.594777 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:21.594781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:21.594795 (+ 14us) server_negotiation.cc:407] Connection header received
0923 19:30:21.594832 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:21.594835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:21.594901 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:21.594977 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:21.595575 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.596262 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.597389 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.597796 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.597895 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.601386 (+ 3491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49089: BlockingRecv error: recv got EOF from 127.0.97.3:49089 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
W20260923 19:30:21.601836 9636 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:21 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:21.684334 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:21.674042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:40861)
0923 19:30:21.674693 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:21.674713 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:21.674865 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:21.675235 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:21.675239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:21.675761 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:21.676005 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.676014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.677129 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.677135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.678268 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:21.678311 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.679224 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.679230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:21.679408 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.680119 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:21.680142 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:21.683992 (+ 3850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":333,"threads_started":1}
W20260923 19:30:21.685027 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:21.674175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40861 (local address 127.0.97.62:46821)
0923 19:30:21.674414 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:21.674419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:21.674818 (+ 399us) server_negotiation.cc:407] Connection header received
0923 19:30:21.674990 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:21.674993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:21.675060 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:21.675152 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:21.676176 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.676934 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.678430 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:21.679005 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:21.679119 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:21.684824 (+ 5705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40861: BlockingRecv error: recv got EOF from 127.0.97.1:40861 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 19:30:22 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:22 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:22.171556 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:22.159212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:45415)
0923 19:30:22.159733 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:22.159749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:22.159881 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:22.160203 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:22.160206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:22.160690 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:22.160897 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:22.160906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.162070 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.162074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:22.163235 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:22.163249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.163967 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.163971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:22.164091 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:22.167983 (+ 3892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:22.168002 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:22.171315 (+ 3313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":238,"threads_started":1}
W20260923 19:30:22.171849 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:22.159288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45415 (local address 127.0.97.62:46821)
0923 19:30:22.159507 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:22.159513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:22.159832 (+ 319us) server_negotiation.cc:407] Connection header received
0923 19:30:22.160008 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:22.160012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:22.160085 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:22.160166 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:22.161069 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.161906 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.163367 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.163796 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.163890 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:22.171566 (+ 7676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45415: BlockingRecv error: recv got EOF from 127.0.97.2:45415 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 19:30:22 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:22 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:22.611387 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:22.602727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:39279)
0923 19:30:22.603316 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:22.603333 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:22.603465 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:22.603762 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:22.603766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:22.604214 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:22.604418 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:22.604428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.605419 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.605423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:22.606394 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:22.606407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.607082 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.607088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:22.607238 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:22.607877 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:22.607893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:22.611075 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":312,"threads_started":1}
W20260923 19:30:22.611505 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:22.602865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39279 (local address 127.0.97.62:46821)
0923 19:30:22.603133 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:22.603140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:22.603410 (+ 270us) server_negotiation.cc:407] Connection header received
0923 19:30:22.603572 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:22.603577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:22.603636 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:22.603711 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:22.604551 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.605281 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.606523 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.606904 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.607018 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:22.611279 (+ 4261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39279: BlockingRecv error: recv got EOF from 127.0.97.3:39279 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 19:30:22 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:22 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:22.694425 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:22.685415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:33709)
0923 19:30:22.686003 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:22.686022 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:22.686155 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:22.686594 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:22.686598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:22.687040 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:22.687224 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:22.687233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.688195 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.688200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:22.689233 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:22.689246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.689958 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.689963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:22.690099 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:22.690851 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:22.690916 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:22.694052 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":330,"threads_started":1}
W20260923 19:30:22.694597 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:22.685445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33709 (local address 127.0.97.62:46821)
0923 19:30:22.685631 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:22.685636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:22.686097 (+ 461us) server_negotiation.cc:407] Connection header received
0923 19:30:22.686310 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:22.686315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:22.686401 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:22.686503 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:22.687342 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.688059 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.689362 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:22.689781 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:22.689871 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:22.694328 (+ 4457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33709: BlockingRecv error: recv got EOF from 127.0.97.1:33709 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 19:30:23 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:23 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:23.181159 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:23.172481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:47403)
0923 19:30:23.173145 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:23.173164 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:23.173321 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:23.173684 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:23.173689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:23.174095 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:23.174241 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:23.174249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.175410 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.175414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:23.176350 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:23.176363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.177096 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.177102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:23.177256 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:23.177998 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:23.178016 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:23.180846 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":364,"threads_started":1}
W20260923 19:30:23.181529 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:23.172614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47403 (local address 127.0.97.62:46821)
0923 19:30:23.172849 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:23.172855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:23.173266 (+ 411us) server_negotiation.cc:407] Connection header received
0923 19:30:23.173438 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:23.173443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:23.173502 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:23.173585 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:23.174451 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.175255 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.176491 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.176927 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.177055 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:23.181205 (+ 4150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47403: BlockingRecv error: recv got EOF from 127.0.97.2:47403 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 23 19:30:23 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:23 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:23.621662 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:23.612439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:54271)
0923 19:30:23.612952 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:23.612974 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:23.613091 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:23.613451 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:23.613454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:23.613879 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:23.614041 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:23.614050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.615279 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.615283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:23.616222 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:23.616234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.617036 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.617041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:23.617158 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:23.617795 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:23.617813 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:23.621356 (+ 3543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
W20260923 19:30:23.621881 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:23.612739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54271 (local address 127.0.97.62:46821)
0923 19:30:23.612952 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:23.612957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:23.613051 (+ 94us) server_negotiation.cc:407] Connection header received
0923 19:30:23.613203 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:23.613208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:23.613307 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:23.613440 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:23.614200 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.615122 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.616386 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.616858 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.617011 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:23.621567 (+ 4556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54271: BlockingRecv error: recv got EOF from 127.0.97.3:54271 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 23 19:30:23 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:23 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:23.704619 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:23.695564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:41301)
0923 19:30:23.696182 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:23.696204 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:23.696368 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:23.696816 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:23.696820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:23.697336 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:23.697540 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:23.697550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.698650 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.698655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:23.699700 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:23.699713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.700406 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.700412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:23.700564 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:23.701222 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:23.701241 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:23.704355 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":297,"threads_started":1}
W20260923 19:30:23.705086 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:23.695769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41301 (local address 127.0.97.62:46821)
0923 19:30:23.695981 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:23.695987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:23.696305 (+ 318us) server_negotiation.cc:407] Connection header received
0923 19:30:23.696521 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:23.696526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:23.696619 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:23.696760 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:23.697727 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.698495 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.699859 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:23.700258 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:23.700352 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:23.704723 (+ 4371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41301: BlockingRecv error: recv got EOF from 127.0.97.1:41301 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 19:30:24 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:24 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:24.191746 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:24.182487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:48349)
0923 19:30:24.183044 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:24.183063 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:24.183200 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:24.183650 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:24.183655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:24.184008 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:24.184214 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:24.184224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.185517 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.185521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:24.186692 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:24.186704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.187536 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.187544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:24.187750 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:24.188420 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:24.188447 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:24.191332 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":294,"threads_started":1}
W20260923 19:30:24.191828 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:24.182556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48349 (local address 127.0.97.62:46821)
0923 19:30:24.182798 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:24.182807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:24.183156 (+ 349us) server_negotiation.cc:407] Connection header received
0923 19:30:24.183311 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:24.183339 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:24.183461 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:24.183587 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:24.184387 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.185364 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.186830 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.187363 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.187497 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:24.191573 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48349: BlockingRecv error: recv got EOF from 127.0.97.2:48349 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 19:30:24 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:24 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:24.631366 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:24.622843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:46101)
0923 19:30:24.623357 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:24.623375 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:24.623540 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:24.623874 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:24.623879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:24.624266 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:24.624447 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:24.624457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.625461 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.625468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:24.626478 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:24.626490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.627173 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.627178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:24.627361 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:24.627999 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:24.628019 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:24.631086 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":285,"threads_started":1}
W20260923 19:30:24.631515 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:24.622923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46101 (local address 127.0.97.62:46821)
0923 19:30:24.623164 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:24.623171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:24.623455 (+ 284us) server_negotiation.cc:407] Connection header received
0923 19:30:24.623654 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:24.623660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:24.623743 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:24.623824 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:24.624586 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.625340 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.626607 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.627038 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.627144 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:24.631265 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46101: BlockingRecv error: recv got EOF from 127.0.97.3:46101 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 19:30:24 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:24 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:24.715211 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:24.705906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:59095)
0923 19:30:24.706572 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:24.706590 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:24.706764 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:24.707239 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:24.707244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:24.707755 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:24.707994 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:24.708007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.709131 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.709136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:24.710187 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:24.710200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.710890 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.710897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:24.711054 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:24.711738 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:24.711758 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:24.714946 (+ 3188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":406,"threads_started":1}
W20260923 19:30:24.715406 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:24.706039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59095 (local address 127.0.97.62:46821)
0923 19:30:24.706394 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:24.706400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:24.706686 (+ 286us) server_negotiation.cc:407] Connection header received
0923 19:30:24.706901 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:24.706906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:24.707053 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:24.707158 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:24.708133 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.708998 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.710347 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:24.710776 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:24.710866 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:24.715162 (+ 4296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59095: BlockingRecv error: recv got EOF from 127.0.97.1:59095 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Sep 23 19:30:25 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:25 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:25.202445 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:25.192999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:34187)
0923 19:30:25.193544 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:25.193561 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:25.193714 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:25.194112 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:25.194116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:25.194599 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:25.194812 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:25.194822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.196211 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.196215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:25.197237 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:25.197250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.197978 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.197983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:25.198149 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:25.198798 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:25.198816 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:25.202152 (+ 3336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":294,"threads_started":1}
W20260923 19:30:25.202684 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:25.193104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34187 (local address 127.0.97.62:46821)
0923 19:30:25.193350 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:25.193358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:25.193655 (+ 297us) server_negotiation.cc:407] Connection header received
0923 19:30:25.193828 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:25.193832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:25.193905 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:25.194046 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:25.194944 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.196067 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.197376 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.197836 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.197933 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:25.202444 (+ 4511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34187: BlockingRecv error: recv got EOF from 127.0.97.2:34187 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 19:30:25 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:25 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:25.641026 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:25.632311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:50501)
0923 19:30:25.632850 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:25.632866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:25.632990 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:25.633454 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:25.633458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:25.633826 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:25.633994 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:25.634003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.635291 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.635295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:25.636263 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:25.636274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.636989 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.636993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:25.637127 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:25.637746 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:25.637764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:25.640798 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":255,"threads_started":1}
W20260923 19:30:25.641341 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:25.632437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50501 (local address 127.0.97.62:46821)
0923 19:30:25.632661 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:25.632666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:25.632943 (+ 277us) server_negotiation.cc:407] Connection header received
0923 19:30:25.633173 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:25.633178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:25.633276 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:25.633379 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:25.634166 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.635127 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.636409 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.636853 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.636960 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:25.641047 (+ 4087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50501: BlockingRecv error: recv got EOF from 127.0.97.3:50501 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 23 19:30:25 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:25 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:25.725329 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:25.716250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:45297)
0923 19:30:25.716892 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:25.716910 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:25.717045 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:25.717364 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:25.717368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:25.717825 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:25.718053 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:25.718063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.719075 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.719079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:25.720076 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:25.720089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.720765 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.720770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:25.720920 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:25.721636 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:25.721655 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:25.724956 (+ 3301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":351,"threads_started":1}
W20260923 19:30:25.725379 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:25.716374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45297 (local address 127.0.97.62:46821)
0923 19:30:25.716592 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:25.716598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:25.716998 (+ 400us) server_negotiation.cc:407] Connection header received
0923 19:30:25.717153 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:25.717156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:25.717222 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:25.717316 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:25.718180 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.718931 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.720213 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:25.720607 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:25.720703 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:25.725169 (+ 4466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45297: BlockingRecv error: recv got EOF from 127.0.97.1:45297 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 19:30:26 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:26 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:26.212720 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:26.203634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:41867)
0923 19:30:26.204202 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:26.204219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:26.204353 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:26.204831 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:26.204836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:26.205264 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:26.205459 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:26.205469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.206690 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.206702 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:26.207643 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:26.207655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.208548 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.208552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:26.208699 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:26.209403 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:26.209422 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:26.212429 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":279,"threads_started":1}
W20260923 19:30:26.213230 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:26.203887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41867 (local address 127.0.97.62:46821)
0923 19:30:26.204126 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:26.204154 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:26.204296 (+ 142us) server_negotiation.cc:407] Connection header received
0923 19:30:26.204526 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:26.204539 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:26.204616 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:26.204745 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:26.205635 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.206526 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.207837 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.208382 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.208496 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:26.212717 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41867: BlockingRecv error: recv got EOF from 127.0.97.2:41867 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 19:30:26 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:26 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:26.651537 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:26.642062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:46835)
0923 19:30:26.642599 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:26.642617 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:26.642750 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:26.643168 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:26.643172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:26.643553 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:26.643754 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:26.643764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.644938 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.644942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:26.645847 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:26.645857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.646789 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.646794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:26.646919 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:26.647598 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:26.647629 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:26.651155 (+ 3526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":250,"threads_started":1}
W20260923 19:30:26.651607 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:26.642190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46835 (local address 127.0.97.62:46821)
0923 19:30:26.642422 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:26.642427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:26.642700 (+ 273us) server_negotiation.cc:407] Connection header received
0923 19:30:26.642910 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:26.642914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:26.643008 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:26.643111 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:26.643974 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.644796 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.646037 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.646633 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.646753 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:26.651363 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46835: BlockingRecv error: recv got EOF from 127.0.97.3:46835 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 19:30:26 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:26 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:26.735229 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:26.726652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:48251)
0923 19:30:26.727193 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:26.727213 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:26.727361 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:26.727705 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:26.727708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:26.728073 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:26.728244 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:26.728252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.729326 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.729331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:26.730325 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:26.730342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.731142 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.731147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:26.731284 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:26.731928 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:26.731951 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:26.735002 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":297,"threads_started":1}
W20260923 19:30:26.735535 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:26.726759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48251 (local address 127.0.97.62:46821)
0923 19:30:26.727012 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:26.727020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:26.727344 (+ 324us) server_negotiation.cc:407] Connection header received
0923 19:30:26.727493 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:26.727498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:26.727575 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:26.727670 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:26.728360 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.729166 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.730478 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:26.730909 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:26.731012 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:26.735264 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48251: BlockingRecv error: recv got EOF from 127.0.97.1:48251 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 23 19:30:27 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:27 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:27.226343 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:27.213785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:59669)
0923 19:30:27.214386 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:27.214403 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:27.214538 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:27.214888 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:27.214893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:27.215273 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:27.215424 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:27.215432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.216397 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.216400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:27.217274 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:27.217284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.217944 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.217950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:27.218101 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:27.222403 (+ 4302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:27.222422 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:27.226006 (+ 3584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":322,"threads_started":1}
W20260923 19:30:27.226606 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:27.213886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59669 (local address 127.0.97.62:46821)
0923 19:30:27.214105 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:27.214111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:27.214489 (+ 378us) server_negotiation.cc:407] Connection header received
0923 19:30:27.214648 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:27.214654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:27.214752 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:27.214837 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:27.215567 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.216269 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.217402 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.217796 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.217903 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:27.226315 (+ 8412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59669: BlockingRecv error: recv got EOF from 127.0.97.2:59669 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 19:30:27 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:27 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:27.661228 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:27.652708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:41507)
0923 19:30:27.653257 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:27.653274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:27.653402 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:27.653745 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:27.653749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:27.654182 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:27.654396 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:27.654409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.655557 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.655562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:27.656512 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:27.656524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.657280 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.657285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:27.657411 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:27.658092 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:27.658110 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:27.660971 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":296,"threads_started":1}
W20260923 19:30:27.661494 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:27.652778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41507 (local address 127.0.97.62:46821)
0923 19:30:27.653011 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:27.653019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:27.653351 (+ 332us) server_negotiation.cc:407] Connection header received
0923 19:30:27.653506 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:27.653510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:27.653591 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:27.653672 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:27.654563 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.655386 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.656654 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.657138 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.657240 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:27.661232 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41507: BlockingRecv error: recv got EOF from 127.0.97.3:41507 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 23 19:30:27 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:27 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:27.745333 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:27.736274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:38995)
0923 19:30:27.737059 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:27.737084 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:27.737267 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:27.737617 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:27.737624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:27.738114 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:27.738351 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:27.738361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.739408 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.739412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:27.740439 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:27.740453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.741156 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.741161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:27.741291 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:27.741961 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:27.741981 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:27.745027 (+ 3046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":404,"threads_started":1}
W20260923 19:30:27.745424 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:27.736413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38995 (local address 127.0.97.62:46821)
0923 19:30:27.736626 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:27.736634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:27.737181 (+ 547us) server_negotiation.cc:407] Connection header received
0923 19:30:27.737399 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:27.737403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:27.737468 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:27.737548 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:27.738475 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.739264 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.740563 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:27.741018 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:27.741109 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:27.745231 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38995: BlockingRecv error: recv got EOF from 127.0.97.1:38995 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 23 19:30:28 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:28 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:28.236946 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:28.227467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:58405)
0923 19:30:28.228156 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:28.228174 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:28.228310 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:28.228681 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:28.228684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:28.229160 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:28.229355 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:28.229365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.230720 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.230725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:28.232182 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:28.232198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.232938 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.232943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:28.233066 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:28.233691 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:28.233709 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:28.236688 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":306,"threads_started":1}
W20260923 19:30:28.237125 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:28.227557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58405 (local address 127.0.97.62:46821)
0923 19:30:28.227800 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:28.227806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:28.228274 (+ 468us) server_negotiation.cc:407] Connection header received
0923 19:30:28.228415 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:28.228419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:28.228514 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:28.228609 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:28.229513 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.230545 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.232320 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.232798 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.232903 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:28.236888 (+ 3985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58405: BlockingRecv error: recv got EOF from 127.0.97.2:58405 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 23 19:30:28 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:28 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:28.675287 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:28.662724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:52385)
0923 19:30:28.663575 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:28.663600 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:28.663801 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:28.664519 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:28.664526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:28.665174 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:28.665788 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:28.665804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.667536 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.667542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:28.669060 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:28.669080 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.670155 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.670163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:28.670427 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:28.671253 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:28.671281 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:28.674890 (+ 3609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":366,"threads_started":1}
W20260923 19:30:28.675381 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:28.662811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52385 (local address 127.0.97.62:46821)
0923 19:30:28.663090 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:28.663098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:28.663701 (+ 603us) server_negotiation.cc:407] Connection header received
0923 19:30:28.663972 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:28.663976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:28.664070 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:28.664166 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:28.666160 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.667372 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.669271 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.669980 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.670115 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:28.675150 (+ 5035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52385: BlockingRecv error: recv got EOF from 127.0.97.3:52385 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 19:30:28 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:28 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:28.755520 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:28.746408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:54913)
0923 19:30:28.746931 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:28.746948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:28.747098 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:28.747400 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:28.747403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:28.747809 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:28.747977 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:28.747985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.749030 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.749034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:28.750018 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:28.750030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.750873 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.750879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:28.751030 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:28.751798 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:28.751817 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:28.755142 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":289,"threads_started":1}
W20260923 19:30:28.755658 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:28.746726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54913 (local address 127.0.97.62:46821)
0923 19:30:28.746933 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:28.746938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:28.747019 (+ 81us) server_negotiation.cc:407] Connection header received
0923 19:30:28.747213 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:28.747217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:28.747284 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:28.747370 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:28.748093 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.748899 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.750165 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:28.750697 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:28.750813 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:28.755388 (+ 4575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54913: BlockingRecv error: recv got EOF from 127.0.97.1:54913 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 23 19:30:29 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:29 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:29.246946 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:29.238100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:56381)
0923 19:30:29.238663 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:29.238684 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:29.238827 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:29.239134 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:29.239137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:29.239580 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:29.239751 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:29.239760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.240983 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.240988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:29.242031 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:29.242044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.242894 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.242899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:29.243011 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:29.243678 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:29.243696 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:29.246665 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":310,"threads_started":1}
W20260923 19:30:29.247182 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:29.238269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56381 (local address 127.0.97.62:46821)
0923 19:30:29.238487 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:29.238493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:29.238759 (+ 266us) server_negotiation.cc:407] Connection header received
0923 19:30:29.238920 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:29.238924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:29.239022 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:29.239098 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:29.239894 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.240824 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.242191 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.242727 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.242842 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:29.246899 (+ 4057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56381: BlockingRecv error: recv got EOF from 127.0.97.2:56381 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 19:30:29 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:29 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:29.685271 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:29.676346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:39083)
0923 19:30:29.676805 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:29.676822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:29.676948 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:29.677286 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:29.677289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:29.677696 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:29.677913 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:29.677926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.679335 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.679339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:29.680299 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:29.680311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.681080 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.681085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:29.681207 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:29.681860 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:29.681878 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:29.684962 (+ 3084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
W20260923 19:30:29.685607 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:29.676410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39083 (local address 127.0.97.62:46821)
0923 19:30:29.676665 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:29.676671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:29.677036 (+ 365us) server_negotiation.cc:407] Connection header received
0923 19:30:29.677101 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:29.677105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:29.677180 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:29.677262 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:29.678098 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.679193 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.680437 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.680879 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.681033 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:29.685313 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39083: BlockingRecv error: recv got EOF from 127.0.97.3:39083 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 19:30:29 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:29 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:29.765718 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:29.756731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:53927)
0923 19:30:29.757357 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:29.757375 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:29.757525 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:29.757868 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:29.757873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:29.758369 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:29.758563 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:29.758573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.759606 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.759610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:29.760807 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:29.760823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.761556 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.761563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:29.761718 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:29.762400 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:29.762419 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:29.765423 (+ 3004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":365,"threads_started":1}
W20260923 19:30:29.765883 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:29.756844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53927 (local address 127.0.97.62:46821)
0923 19:30:29.757080 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:29.757087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:29.757463 (+ 376us) server_negotiation.cc:407] Connection header received
0923 19:30:29.757657 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:29.757663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:29.757731 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:29.757815 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:29.758695 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.759488 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.760944 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:29.761402 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:29.761529 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:29.765694 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53927: BlockingRecv error: recv got EOF from 127.0.97.1:53927 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 23 19:30:30 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:30 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:30.256812 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:30.247836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:46535)
0923 19:30:30.248489 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:30.248506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:30.248636 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:30.249038 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:30.249043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:30.249520 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:30.249710 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:30.249719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.250847 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.250851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:30.251846 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:30.251858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.252505 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.252510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:30.252650 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:30.253295 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:30.253313 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:30.256538 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":372,"threads_started":1}
W20260923 19:30:30.257113 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:30.247961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46535 (local address 127.0.97.62:46821)
0923 19:30:30.248185 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:30.248190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:30.248601 (+ 411us) server_negotiation.cc:407] Connection header received
0923 19:30:30.248742 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:30.248746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:30.248815 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:30.248899 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:30.249878 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.250735 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.251968 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.252372 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.252462 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:30.256792 (+ 4330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46535: BlockingRecv error: recv got EOF from 127.0.97.2:46535 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 23 19:30:30 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:30 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:30.696061 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:30.686515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:45743)
0923 19:30:30.687200 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:30.687221 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:30.687404 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:30.687771 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:30.687776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:30.688257 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:30.688488 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:30.688499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.689574 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.689581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:30.690647 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:30.690660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.691474 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.691480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:30.691664 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:30.692312 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:30.692332 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:30.695798 (+ 3466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":343,"threads_started":1}
W20260923 19:30:30.696234 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:30.686629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45743 (local address 127.0.97.62:46821)
0923 19:30:30.686877 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:30.686884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:30.687333 (+ 449us) server_negotiation.cc:407] Connection header received
0923 19:30:30.687542 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:30.687548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:30.687619 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:30.687708 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:30.688639 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.689419 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.690789 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.691293 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.691420 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:30.696008 (+ 4588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45743: BlockingRecv error: recv got EOF from 127.0.97.3:45743 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 23 19:30:30 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:30 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:30.775854 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:30.766862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:55077)
0923 19:30:30.767412 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:30.767432 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:30.767565 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:30.767960 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:30.767964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:30.768358 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:30.768528 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:30.768537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.769703 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.769707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:30.770783 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:30.770796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.771586 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.771592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:30.771762 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:30.772372 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:30.772390 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:30.775611 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":311,"threads_started":1}
W20260923 19:30:30.776204 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:30.766935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55077 (local address 127.0.97.62:46821)
0923 19:30:30.767174 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:30.767183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:30.767536 (+ 353us) server_negotiation.cc:407] Connection header received
0923 19:30:30.767703 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:30.767711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:30.767786 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:30.767881 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:30.768690 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.769568 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.770905 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:30.771407 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:30.771539 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:30.775857 (+ 4318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55077: BlockingRecv error: recv got EOF from 127.0.97.1:55077 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 23 19:30:31 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:31 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:31.266863 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:31.257794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:54543)
0923 19:30:31.258395 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:31.258412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:31.258556 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:31.258891 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:31.258894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:31.259247 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:31.259435 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:31.259445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.260496 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.260499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:31.261456 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:31.261468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.262147 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.262151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:31.262270 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:31.262917 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:31.262934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:31.266532 (+ 3598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":275,"threads_started":1}
W20260923 19:30:31.267086 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:31.257910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54543 (local address 127.0.97.62:46821)
0923 19:30:31.258084 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:31.258089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:31.258483 (+ 394us) server_negotiation.cc:407] Connection header received
0923 19:30:31.258673 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:31.258677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:31.258747 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:31.258858 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:31.259554 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.260386 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.261596 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.262008 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.262115 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:31.266794 (+ 4679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54543: BlockingRecv error: recv got EOF from 127.0.97.2:54543 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 19:30:31 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:31 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:31.706075 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:31.697146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:36175)
0923 19:30:31.697682 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:31.697701 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:31.697842 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:31.698199 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:31.698205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:31.698657 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:31.698818 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:31.698828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.699982 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.699987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:31.700952 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:31.700964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.701662 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.701668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:31.701836 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:31.702575 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:31.702594 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:31.705806 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":304,"threads_started":1}
W20260923 19:30:31.706313 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:31.697309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36175 (local address 127.0.97.62:46821)
0923 19:30:31.697502 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:31.697509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:31.697795 (+ 286us) server_negotiation.cc:407] Connection header received
0923 19:30:31.697957 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:31.697963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:31.698032 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:31.698111 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:31.698971 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.699840 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.701086 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.701521 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.701637 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:31.706029 (+ 4392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36175: BlockingRecv error: recv got EOF from 127.0.97.3:36175 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 23 19:30:31 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:31 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:31.785842 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:31.776915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:56161)
0923 19:30:31.777484 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:31.777515 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:31.777660 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:31.777969 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:31.777974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:31.778505 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:31.778707 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:31.778716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.779761 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.779766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:31.780700 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:31.780711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.781454 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.781460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:31.781638 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:31.782370 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:31.782390 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:31.785556 (+ 3166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":323,"threads_started":1}
W20260923 19:30:31.785986 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:31.777076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56161 (local address 127.0.97.62:46821)
0923 19:30:31.777308 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:31.777322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:31.777598 (+ 276us) server_negotiation.cc:407] Connection header received
0923 19:30:31.777779 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:31.777784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:31.777850 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:31.777929 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:31.778839 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.779634 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.780826 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:31.781304 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:31.781416 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:31.785791 (+ 4375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56161: BlockingRecv error: recv got EOF from 127.0.97.1:56161 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 19:30:32 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:32 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:32.282503 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:32.267887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:41721)
0923 19:30:32.268461 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:32.268478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:32.268611 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:32.268957 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:32.268961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:32.269470 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:32.269714 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:32.269725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.271020 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.271026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:32.272111 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:32.272128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.272992 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.272997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:32.273175 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:32.279132 (+ 5957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:32.279156 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:32.282181 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":331,"threads_started":1}
W20260923 19:30:32.282737 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:32.268023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41721 (local address 127.0.97.62:46821)
0923 19:30:32.268236 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:32.268241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:32.268579 (+ 338us) server_negotiation.cc:407] Connection header received
0923 19:30:32.268716 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:32.268720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:32.268789 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:32.268875 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:32.269903 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.270828 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.272253 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.272822 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.272955 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:32.282472 (+ 9517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41721: BlockingRecv error: recv got EOF from 127.0.97.2:41721 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 23 19:30:32 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:32 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:32.715909 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:32.707121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:47733)
0923 19:30:32.707706 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:32.707727 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:32.707860 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:32.708147 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:32.708150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:32.708605 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:32.708867 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:32.708878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.709931 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.709935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:32.710924 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:32.710936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.711585 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.711589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:32.711720 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:32.712401 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:32.712420 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:32.715613 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":311,"threads_started":1}
W20260923 19:30:32.716034 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:32.707246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47733 (local address 127.0.97.62:46821)
0923 19:30:32.707424 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:32.707429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:32.707799 (+ 370us) server_negotiation.cc:407] Connection header received
0923 19:30:32.707962 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:32.707966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:32.708036 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:32.708120 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:32.708991 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.709802 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.711046 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.711464 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.711557 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:32.715835 (+ 4278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47733: BlockingRecv error: recv got EOF from 127.0.97.3:47733 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 19:30:32 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:32 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:32.795683 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:32.787035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:54851)
0923 19:30:32.787616 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:32.787638 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:32.787810 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:32.788189 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:32.788193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:32.788588 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:32.788777 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:32.788785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.789912 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.789916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:32.790890 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:32.790901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.791721 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.791726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:32.791868 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:32.792514 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:32.792532 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:32.795419 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":282,"threads_started":1}
W20260923 19:30:32.795845 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:32.787112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54851 (local address 127.0.97.62:46821)
0923 19:30:32.787334 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:32.787342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:32.787734 (+ 392us) server_negotiation.cc:407] Connection header received
0923 19:30:32.787941 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:32.787947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:32.788034 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:32.788148 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:32.788923 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.789777 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.791017 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:32.791574 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:32.791679 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:32.795620 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54851: BlockingRecv error: recv got EOF from 127.0.97.1:54851 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 19:30:33 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:33 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:33.292559 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:33.283605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:34729)
0923 19:30:33.284203 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:33.284220 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:33.284376 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:33.284717 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:33.284721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:33.285225 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:33.285395 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:33.285404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.286523 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.286527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:33.287472 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:33.287483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.288191 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.288197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:33.288363 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:33.289040 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:33.289082 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:33.292213 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":332,"threads_started":1}
W20260923 19:30:33.292714 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:33.283713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34729 (local address 127.0.97.62:46821)
0923 19:30:33.283923 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:33.283931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:33.284315 (+ 384us) server_negotiation.cc:407] Connection header received
0923 19:30:33.284502 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:33.284506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:33.284579 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:33.284661 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:33.285529 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.286385 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.287610 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.288040 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.288150 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:33.292440 (+ 4290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34729: BlockingRecv error: recv got EOF from 127.0.97.2:34729 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 19:30:33 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:33 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:33.726166 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:33.716954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:37851)
0923 19:30:33.717525 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:33.717543 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:33.717688 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:33.718137 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:33.718142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:33.718604 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:33.718785 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:33.718794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.719983 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.719987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:33.721011 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:33.721025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.721834 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.721838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:33.721986 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:33.722766 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:33.722785 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:33.725809 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":305,"threads_started":1}
W20260923 19:30:33.726377 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:33.717022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37851 (local address 127.0.97.62:46821)
0923 19:30:33.717235 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:33.717243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:33.717634 (+ 391us) server_negotiation.cc:407] Connection header received
0923 19:30:33.717836 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:33.717840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:33.717916 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:33.718064 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:33.718952 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.719808 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.721171 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.721672 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.721780 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:33.726098 (+ 4318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37851: BlockingRecv error: recv got EOF from 127.0.97.3:37851 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 23 19:30:33 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:33 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:33.805684 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:33.796815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:43297)
0923 19:30:33.797423 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:33.797440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:33.797602 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:33.797940 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:33.797945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:33.798442 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:33.798618 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:33.798628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.799672 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.799676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:33.800647 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:33.800660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.801408 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.801414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:33.801569 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:33.802276 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:33.802328 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:33.805388 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":350,"threads_started":1}
W20260923 19:30:33.805812 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:33.796934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43297 (local address 127.0.97.62:46821)
0923 19:30:33.797184 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:33.797191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:33.797549 (+ 358us) server_negotiation.cc:407] Connection header received
0923 19:30:33.797732 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:33.797737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:33.797805 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:33.797887 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:33.798757 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.799542 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.800781 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:33.801226 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:33.801333 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:33.805611 (+ 4278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43297: BlockingRecv error: recv got EOF from 127.0.97.1:43297 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 23 19:30:34 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:34 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:34.302739 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:34.293826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:51001)
0923 19:30:34.294413 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:34.294431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:34.294574 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:34.294894 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:34.294897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:34.295325 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:34.295601 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:34.295612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.296706 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.296710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:34.297670 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:34.297683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.298652 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.298657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:34.298848 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:34.299557 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:34.299581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:34.302465 (+ 2884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":298,"threads_started":1}
W20260923 19:30:34.302971 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:34.294004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51001 (local address 127.0.97.62:46821)
0923 19:30:34.294223 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:34.294228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:34.294519 (+ 291us) server_negotiation.cc:407] Connection header received
0923 19:30:34.294677 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:34.294681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:34.294768 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:34.294858 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:34.295757 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.296572 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.297822 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.298489 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.298605 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:34.302708 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51001: BlockingRecv error: recv got EOF from 127.0.97.2:51001 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 23 19:30:34 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:34 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:34.736152 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:34.727218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:32825)
0923 19:30:34.727720 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:34.727737 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:34.727874 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:34.728192 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:34.728196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:34.728629 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:34.728824 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:34.728834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.729950 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.729955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:34.731065 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:34.731077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.731800 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.731804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:34.731925 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:34.732586 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:34.732604 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:34.735882 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":255,"threads_started":1}
W20260923 19:30:34.736254 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:34.727302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32825 (local address 127.0.97.62:46821)
0923 19:30:34.727578 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:34.727584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:34.727841 (+ 257us) server_negotiation.cc:407] Connection header received
0923 19:30:34.727974 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:34.727978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:34.728047 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:34.728146 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:34.728960 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.729803 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.731209 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.731632 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.731732 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:34.736047 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:32825: BlockingRecv error: recv got EOF from 127.0.97.3:32825 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 19:30:34 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:34 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:34.815773 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:34.806886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:33501)
0923 19:30:34.807454 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:34.807471 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:34.807632 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:34.807944 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:34.807947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:34.808441 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:34.808689 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:34.808703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.810030 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.810033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:34.811135 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:34.811149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.811859 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.811863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:34.811984 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:34.812634 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:34.812653 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:34.815474 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":332,"threads_started":1}
W20260923 19:30:34.815932 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:34.807003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33501 (local address 127.0.97.62:46821)
0923 19:30:34.807312 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:34.807319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:34.807572 (+ 253us) server_negotiation.cc:407] Connection header received
0923 19:30:34.807750 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:34.807754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:34.807829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:34.807922 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:34.808842 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.809893 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.811277 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:34.811726 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:34.811826 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:34.815686 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33501: BlockingRecv error: recv got EOF from 127.0.97.1:33501 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 23 19:30:35 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:35 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:35.313352 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:35.303754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:52545)
0923 19:30:35.304455 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:35.304476 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:35.304638 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:35.305025 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:35.305029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:35.305413 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:35.305587 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:35.305595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.306752 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.306757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:35.308013 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:35.308029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.308898 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.308905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:35.309156 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:35.309890 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:35.309908 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:35.312999 (+ 3091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":396,"threads_started":1}
W20260923 19:30:35.313490 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:35.303826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52545 (local address 127.0.97.62:46821)
0923 19:30:35.304033 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:35.304040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:35.304576 (+ 536us) server_negotiation.cc:407] Connection header received
0923 19:30:35.304769 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:35.304776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:35.304863 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:35.304962 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:35.305723 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.306583 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.308154 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.308731 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.308870 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:35.313264 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52545: BlockingRecv error: recv got EOF from 127.0.97.2:52545 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 23 19:30:35 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:35 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:35.746567 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:35.737419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.3:42807)
0923 19:30:35.737991 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:35.738012 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:35.738179 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:35.738577 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:35.738583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:35.739041 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:35.739220 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:35.739231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.740379 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.740387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:35.741404 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:35.741418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.742173 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.742180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:35.742407 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:35.743096 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:35.743114 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:35.746169 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":309,"threads_started":1}
W20260923 19:30:35.746763 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:35.737603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42807 (local address 127.0.97.62:46821)
0923 19:30:35.737819 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:35.737825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:35.738118 (+ 293us) server_negotiation.cc:407] Connection header received
0923 19:30:35.738330 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:35.738337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:35.738415 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:35.738507 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:35.739381 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.740222 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.741553 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.742026 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.742147 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:35.746511 (+ 4364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42807: BlockingRecv error: recv got EOF from 127.0.97.3:42807 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 19:30:35 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:35 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:35.826005 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:35.816971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.1:37009)
0923 19:30:35.817453 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:35.817473 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:35.817594 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:35.817868 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:35.817871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:35.818344 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:35.818562 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:35.818572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.819691 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.819695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:35.820714 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:35.820727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.821539 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.821544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:35.821670 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:35.822427 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:35.822447 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:35.825613 (+ 3166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
W20260923 19:30:35.826090 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:35.817059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37009 (local address 127.0.97.62:46821)
0923 19:30:35.817285 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:35.817292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:35.817545 (+ 253us) server_negotiation.cc:407] Connection header received
0923 19:30:35.817689 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:35.817693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:35.817759 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:35.817838 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:35.818715 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.819512 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.820851 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:35.821381 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:35.821483 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:35.825864 (+ 4381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37009: BlockingRecv error: recv got EOF from 127.0.97.1:37009 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 19:30:36 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 19:30:36 dist-test-slave-kxfb krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 19:30:36.324368 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:36.314994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46821 (local address 127.0.97.2:56375)
0923 19:30:36.315592 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:36.315618 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:36.315778 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:36.316109 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:36.316114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:36.316692 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:36.316947 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:36.316962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.318430 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.318436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:36.319612 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:36.319625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.320395 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.320400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:36.320554 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:36.321178 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:36.321196 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:36.324043 (+ 2847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46821: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":344,"threads_started":1}
W20260923 19:30:36.324467 9375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:36.315143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56375 (local address 127.0.97.62:46821)
0923 19:30:36.315365 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:36.315372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:36.315705 (+ 333us) server_negotiation.cc:407] Connection header received
0923 19:30:36.315908 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:36.315913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:36.315981 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:36.316064 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:36.317085 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.318250 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.319747 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.320240 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.320357 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:36.324255 (+ 3898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56375: BlockingRecv error: recv got EOF from 127.0.97.2:56375 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 19:30:36.576912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9225
I20260923 19:30:36.590041 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:36.578216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.0.97.1:33039)
0923 19:30:36.578851 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:36.578860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:36.578895 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:30:36.578972 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:36.578981 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:36.579278 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:36.579467 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:36.581019 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.582168 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.583403 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.584054 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.584177 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:36.586872 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:36.586882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:36.586899 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:36.586940 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:36.589479 (+ 2539us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 19:30:36.589777 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33052: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":325,"threads_started":1}
W20260923 19:30:36.590147 9686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33052: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 19:30:36.590195 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9225: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33039: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 19:30:36.590409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 9225
I20260923 19:30:36.606941 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9377
I20260923 19:30:36.620016 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:36.608148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.0.97.2:33575)
0923 19:30:36.608810 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:36.608819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:36.608854 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:30:36.608925 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:36.608933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:36.609217 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:36.609409 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:36.610936 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.612086 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.613333 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.613958 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.614076 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:36.616671 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:36.616681 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:36.616701 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:36.616738 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:36.619328 (+ 2590us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 19:30:36.619665 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35000: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":308,"threads_started":1}
W20260923 19:30:36.620118 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9377: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:33575: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 19:30:36.620148 9687 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35000: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 19:30:36.620301 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 9377
I20260923 19:30:36.636001 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9508
I20260923 19:30:36.649240 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:36.637199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.0.97.3:44805)
0923 19:30:36.637875 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:36.637883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:36.637914 (+ 31us) server_negotiation.cc:407] Connection header received
0923 19:30:36.637994 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:36.638002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:36.638281 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:36.638505 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:36.639964 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.641116 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.642466 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:36.643102 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:36.643212 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:36.646033 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:36.646043 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:36.646061 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:36.646138 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:36.648684 (+ 2546us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 19:30:36.648960 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41950: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":390,"threads_started":1}
W20260923 19:30:36.649346 9688 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41950: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 19:30:36.649575 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9508: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:44805: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 19:30:36.649726 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 9508
I20260923 19:30:36.665275 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9156
I20260923 19:30:36.666946 9218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:36.785274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 9156
2026-09-23T19:30:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21439 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T19:30:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:30:36Z Disabled control of system clock
I20260923 19:30:36.869153 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46671
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 19:30:37.248449 9693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:37.248843 9693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:37.248965 9693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:37.257022 9693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:30:37.257187 9693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:37.257239 9693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:37.257324 9693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:30:37.257385 9693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:30:37.267679 9693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:37.269407 9693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:37.271610 9693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:37.285521 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:37.285635 9698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:37.285557 9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:37.286347 9693 server_base.cc:1034] running on GCE node
I20260923 19:30:37.287317 9693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:37.289026 9693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:37.290273 9693 hybrid_clock.cc:648] HybridClock initialized: now 1790191837290208 us; error 45 us; skew 500 ppm
I20260923 19:30:37.290750 9693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:30:37.302744 9693 webserver.cc:460] Webserver started at http://127.0.97.62:37437/ using document root <none> and password file <none>
I20260923 19:30:37.303694 9693 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:37.303818 9693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:37.304179 9693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:37.307787 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "2e39da4a7f844a4a9614948898da1e6e"
format_stamp: "Formatted at 2026-09-23 19:30:37 on dist-test-slave-kxfb"
server_key: "fce52840024c4a77464ca3df2cb63159"
server_key_iv: "505bca0106fb7dff6902f48426184f64"
server_key_version: "encryptionkey@0"
I20260923 19:30:37.308668 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "2e39da4a7f844a4a9614948898da1e6e"
format_stamp: "Formatted at 2026-09-23 19:30:37 on dist-test-slave-kxfb"
server_key: "fce52840024c4a77464ca3df2cb63159"
server_key_iv: "505bca0106fb7dff6902f48426184f64"
server_key_version: "encryptionkey@0"
I20260923 19:30:37.314937 9693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260923 19:30:37.319114 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:37.321352 9693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:30:37.321614 9693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "2e39da4a7f844a4a9614948898da1e6e"
format_stamp: "Formatted at 2026-09-23 19:30:37 on dist-test-slave-kxfb"
server_key: "fce52840024c4a77464ca3df2cb63159"
server_key_iv: "505bca0106fb7dff6902f48426184f64"
server_key_version: "encryptionkey@0"
I20260923 19:30:37.321895 9693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:37.350962 9693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:37.352035 9693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:37.352345 9693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:37.372848 9693 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46671
I20260923 19:30:37.372874 9758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46671 every 8 connection(s)
I20260923 19:30:37.374562 9693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:30:37.377394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 9693
I20260923 19:30:37.377604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:30:37.378391 388 external_mini_cluster.cc:1428] Setting key d6cf026a2866605d6c6689f5069c1b73
I20260923 19:30:37.378934 9759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:37.393285 9759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e: Bootstrap starting.
I20260923 19:30:37.394680 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:37.380749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.0.97.62:46671)
0923 19:30:37.381930 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:37.381944 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:37.382006 (+ 62us) server_negotiation.cc:407] Connection header received
0923 19:30:37.383839 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:37.383896 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:37.384539 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:37.385023 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:37.385795 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:37.387333 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:37.389225 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:37.390090 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:37.390277 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:37.391733 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:37.391742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:37.391762 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:37.392456 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:37.392559 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:37.392593 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:37.392696 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:37.392950 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 19:30:37.393362 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":376,"threads_started":1}
I20260923 19:30:37.397030 9759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:37.398146 9759 log.cc:824] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:37.401185 9759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e: No bootstrap required, opened a new log
I20260923 19:30:37.408399 9759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39da4a7f844a4a9614948898da1e6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46671 } }
I20260923 19:30:37.408802 9759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:37.408910 9759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e39da4a7f844a4a9614948898da1e6e, State: Initialized, Role: FOLLOWER
I20260923 19:30:37.409502 9759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39da4a7f844a4a9614948898da1e6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46671 } }
I20260923 19:30:37.409710 9759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:30:37.409845 9759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:30:37.409992 9759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:37.412209 9759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39da4a7f844a4a9614948898da1e6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46671 } }
I20260923 19:30:37.412693 9759 leader_election.cc:302] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e39da4a7f844a4a9614948898da1e6e; no voters:
I20260923 19:30:37.413339 9759 leader_election.cc:288] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:30:37.413574 9764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:37.414371 9764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [term 1 LEADER]: Becoming Leader. State: Replica: 2e39da4a7f844a4a9614948898da1e6e, State: Running, Role: LEADER
I20260923 19:30:37.415067 9764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39da4a7f844a4a9614948898da1e6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46671 } }
I20260923 19:30:37.415499 9759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:30:37.418984 9765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e39da4a7f844a4a9614948898da1e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39da4a7f844a4a9614948898da1e6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46671 } } }
I20260923 19:30:37.419229 9766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e39da4a7f844a4a9614948898da1e6e. Latest consensus state: current_term: 1 leader_uuid: "2e39da4a7f844a4a9614948898da1e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39da4a7f844a4a9614948898da1e6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46671 } } }
I20260923 19:30:37.419497 9765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:37.419519 9766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:37.420535 9773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:30:37.425751 9773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:30:37.436056 9773 catalog_manager.cc:1347] Generated new cluster ID: 6338472c64be4361adaf0c7c451357e7
I20260923 19:30:37.436177 9773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:30:37.462548 9773 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:30:37.464164 9773 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:30:37.473526 9773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e39da4a7f844a4a9614948898da1e6e: Generated new TSK 0
I20260923 19:30:37.474593 9773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:30:37.483323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46671
--builtin_ntp_servers=127.0.97.20:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 19:30:37.872463 9783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:37.872819 9783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:37.872934 9783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:37.880755 9783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:37.880872 9783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:37.881016 9783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:30:37.890772 9783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:37.892375 9783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:37.894474 9783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:37.909313 9788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:37.909261 9791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:37.909341 9789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:37.910585 9783 server_base.cc:1034] running on GCE node
I20260923 19:30:37.911298 9783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:37.912417 9783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:37.913635 9783 hybrid_clock.cc:648] HybridClock initialized: now 1790191837913541 us; error 80 us; skew 500 ppm
I20260923 19:30:37.913990 9783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:30:37.916759 9783 webserver.cc:460] Webserver started at http://127.0.97.1:35091/ using document root <none> and password file <none>
I20260923 19:30:37.917619 9783 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:37.917747 9783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:37.918114 9783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:37.921411 9783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "adfea615546d45cf9f2f8a36990c6ae9"
format_stamp: "Formatted at 2026-09-23 19:30:37 on dist-test-slave-kxfb"
server_key: "8e11e79df7b16fc3fad545c156284031"
server_key_iv: "27f5883f04b37fc1b9206acecd9641ac"
server_key_version: "encryptionkey@0"
I20260923 19:30:37.922245 9783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "adfea615546d45cf9f2f8a36990c6ae9"
format_stamp: "Formatted at 2026-09-23 19:30:37 on dist-test-slave-kxfb"
server_key: "8e11e79df7b16fc3fad545c156284031"
server_key_iv: "27f5883f04b37fc1b9206acecd9641ac"
server_key_version: "encryptionkey@0"
I20260923 19:30:37.928131 9783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 19:30:37.932138 9797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:37.933769 9783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 19:30:37.933929 9783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "adfea615546d45cf9f2f8a36990c6ae9"
format_stamp: "Formatted at 2026-09-23 19:30:37 on dist-test-slave-kxfb"
server_key: "8e11e79df7b16fc3fad545c156284031"
server_key_iv: "27f5883f04b37fc1b9206acecd9641ac"
server_key_version: "encryptionkey@0"
I20260923 19:30:37.934175 9783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:37.960469 9783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:37.961488 9783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:37.961773 9783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:37.962927 9783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:37.964681 9783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:37.964771 9783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:37.964908 9783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:37.964996 9783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:37.999536 9783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35981
I20260923 19:30:37.999645 9909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35981 every 8 connection(s)
I20260923 19:30:38.001086 9783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:30:38.003716 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 9783
I20260923 19:30:38.003935 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:30:38.004706 388 external_mini_cluster.cc:1428] Setting key a43bcdb7dd9b45e9d0ff6feb7c026a1b
I20260923 19:30:38.007663 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46671
--builtin_ntp_servers=127.0.97.20:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 19:30:38.014945 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:38.003947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51739 (local address 127.0.97.62:46671)
0923 19:30:38.004489 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:38.004497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:38.005744 (+ 1247us) server_negotiation.cc:407] Connection header received
0923 19:30:38.007595 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:38.007601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:38.007708 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:38.007844 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:38.009030 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.010048 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.011734 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.012291 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.012410 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:38.013977 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:38.013984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:38.013989 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:38.014013 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:38.014065 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:38.014080 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:38.014175 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:38.014443 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 19:30:38.014555 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":244,"threads_started":1}
I20260923 19:30:38.015910 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:38.004484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46671 (local address 127.0.97.1:51739)
0923 19:30:38.005570 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:38.005617 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:38.007314 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:38.008133 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:38.008142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:38.008241 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:38.008850 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:38.008867 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.010234 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.010238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:38.011585 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:38.011597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.012512 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.012517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:38.012722 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:38.013607 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:30:38.013609 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:38.013650 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:30:38.013738 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:38.014195 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:38.014204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:38.014210 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 19:30:38.014423 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 19:30:38.014737 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":322,"threads_started":1}
I20260923 19:30:38.017968 9910 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46671
I20260923 19:30:38.018337 9910 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:38.019272 9910 heartbeater.cc:511] Master 127.0.97.62:46671 requested a full tablet report, sending...
I20260923 19:30:38.021719 9724 ts_manager.cc:194] Registered new tserver with Master: adfea615546d45cf9f2f8a36990c6ae9 (127.0.97.1:35981)
I20260923 19:30:38.023648 9724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:51739
W20260923 19:30:38.347352 9914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:38.347630 9914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:38.347709 9914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:38.354774 9914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:38.354874 9914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:38.354977 9914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:30:38.363762 9914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:38.364996 9914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:38.366694 9914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:38.380554 9919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:38.380525 9920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:38.381338 9922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:38.382053 9914 server_base.cc:1034] running on GCE node
I20260923 19:30:38.382894 9914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:38.384148 9914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:38.385401 9914 hybrid_clock.cc:648] HybridClock initialized: now 1790191838385325 us; error 62 us; skew 500 ppm
I20260923 19:30:38.385812 9914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:30:38.388640 9914 webserver.cc:460] Webserver started at http://127.0.97.2:40119/ using document root <none> and password file <none>
I20260923 19:30:38.389513 9914 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:38.389621 9914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:38.389968 9914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:38.393226 9914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "0bef71554e504639b58a2d0ca110d383"
format_stamp: "Formatted at 2026-09-23 19:30:38 on dist-test-slave-kxfb"
server_key: "c82f4b896fb5d487598d652a34ef5030"
server_key_iv: "aeb7b6e7f535190483b3f84b7ca80d6c"
server_key_version: "encryptionkey@0"
I20260923 19:30:38.394042 9914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0bef71554e504639b58a2d0ca110d383"
format_stamp: "Formatted at 2026-09-23 19:30:38 on dist-test-slave-kxfb"
server_key: "c82f4b896fb5d487598d652a34ef5030"
server_key_iv: "aeb7b6e7f535190483b3f84b7ca80d6c"
server_key_version: "encryptionkey@0"
I20260923 19:30:38.399856 9914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 19:30:38.403911 9928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:38.405563 9914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:30:38.405773 9914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "0bef71554e504639b58a2d0ca110d383"
format_stamp: "Formatted at 2026-09-23 19:30:38 on dist-test-slave-kxfb"
server_key: "c82f4b896fb5d487598d652a34ef5030"
server_key_iv: "aeb7b6e7f535190483b3f84b7ca80d6c"
server_key_version: "encryptionkey@0"
I20260923 19:30:38.406034 9914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:38.428403 9914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:38.429437 9914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:38.429721 9914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:38.430862 9914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:38.432570 9914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:38.432668 9914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:38.432781 9914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:38.432852 9914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:38.466575 9914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39939
I20260923 19:30:38.466658 10040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39939 every 8 connection(s)
I20260923 19:30:38.468034 9914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:30:38.477618 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 9914
I20260923 19:30:38.477814 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:30:38.478262 388 external_mini_cluster.cc:1428] Setting key e20561a3459ffead73a74f001ec57a1a
I20260923 19:30:38.480834 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:38.470870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33235 (local address 127.0.97.62:46671)
0923 19:30:38.471039 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:38.471043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:38.472616 (+ 1573us) server_negotiation.cc:407] Connection header received
0923 19:30:38.474545 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:38.474551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:38.474629 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:38.474712 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:38.475734 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.476443 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.478063 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.478537 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.478648 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:38.480046 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:38.480053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:38.480057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:38.480076 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:38.480107 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:38.480120 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:38.480194 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:38.480407 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 19:30:38.480504 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 19:30:38.481464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46671
--builtin_ntp_servers=127.0.97.20:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 19:30:38.481943 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:38.471413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46671 (local address 127.0.97.2:33235)
0923 19:30:38.472447 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:38.472493 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:38.474250 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:38.474937 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:38.474944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:38.475026 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:38.475583 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:38.475598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.476580 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.476584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:38.477943 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:38.477955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.478747 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.478755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:38.479046 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:38.479734 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:30:38.479737 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:38.479775 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:30:38.479867 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:38.480201 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:38.480211 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:38.480219 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 19:30:38.480378 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 19:30:38.480785 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":346,"threads_started":1}
I20260923 19:30:38.483799 10041 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46671
I20260923 19:30:38.484146 10041 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:38.484917 10041 heartbeater.cc:511] Master 127.0.97.62:46671 requested a full tablet report, sending...
I20260923 19:30:38.486513 9724 ts_manager.cc:194] Registered new tserver with Master: 0bef71554e504639b58a2d0ca110d383 (127.0.97.2:39939)
I20260923 19:30:38.487615 9724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:33235
W20260923 19:30:38.845535 10044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:38.845824 10044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:38.845903 10044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:38.852967 10044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:38.853075 10044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:38.853180 10044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:30:38.862123 10044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:38.863634 10044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:38.865486 10044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:38.879503 10049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:38.879470 10052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:38.879408 10050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:38.880400 10044 server_base.cc:1034] running on GCE node
I20260923 19:30:38.881153 10044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:38.882318 10044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:38.883643 10044 hybrid_clock.cc:648] HybridClock initialized: now 1790191838883516 us; error 48 us; skew 500 ppm
I20260923 19:30:38.884080 10044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:30:38.887403 10044 webserver.cc:460] Webserver started at http://127.0.97.3:37681/ using document root <none> and password file <none>
I20260923 19:30:38.888294 10044 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:38.888418 10044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:38.888810 10044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:38.892386 10044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "2ea500cd929f4b4cb9e422a55c7b2a00"
format_stamp: "Formatted at 2026-09-23 19:30:38 on dist-test-slave-kxfb"
server_key: "7e6d1be614553fd4ec70e5ceeeb495f7"
server_key_iv: "7f809bc58d1da4eded86e3b1a7ae63f7"
server_key_version: "encryptionkey@0"
I20260923 19:30:38.893359 10044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2ea500cd929f4b4cb9e422a55c7b2a00"
format_stamp: "Formatted at 2026-09-23 19:30:38 on dist-test-slave-kxfb"
server_key: "7e6d1be614553fd4ec70e5ceeeb495f7"
server_key_iv: "7f809bc58d1da4eded86e3b1a7ae63f7"
server_key_version: "encryptionkey@0"
I20260923 19:30:38.899895 10044 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260923 19:30:38.904420 10058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:38.906255 10044 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 19:30:38.906486 10044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "2ea500cd929f4b4cb9e422a55c7b2a00"
format_stamp: "Formatted at 2026-09-23 19:30:38 on dist-test-slave-kxfb"
server_key: "7e6d1be614553fd4ec70e5ceeeb495f7"
server_key_iv: "7f809bc58d1da4eded86e3b1a7ae63f7"
server_key_version: "encryptionkey@0"
I20260923 19:30:38.906733 10044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:38.938361 10044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:38.939493 10044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:38.939775 10044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:38.941053 10044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:38.943003 10044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:38.943110 10044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:38.943233 10044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:38.943307 10044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:38.977785 10044 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35575
I20260923 19:30:38.977842 10170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35575 every 8 connection(s)
I20260923 19:30:38.979336 10044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:30:38.989514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 10044
I20260923 19:30:38.989689 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:30:38.990226 388 external_mini_cluster.cc:1428] Setting key 544731cc3e7f15fec65acfe4c49ebfdd
I20260923 19:30:38.993157 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:38.982072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37037 (local address 127.0.97.62:46671)
0923 19:30:38.982616 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:38.982622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:38.983974 (+ 1352us) server_negotiation.cc:407] Connection header received
0923 19:30:38.985929 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:38.985933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:38.986008 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:38.986130 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:38.987392 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.988128 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.989844 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.990449 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.990585 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:38.992138 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:38.992144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:38.992148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:38.992170 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:38.992241 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:38.992260 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:38.992360 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:38.992685 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 19:30:38.992791 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":275,"threads_started":1}
I20260923 19:30:38.994338 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:38.982671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46671 (local address 127.0.97.3:37037)
0923 19:30:38.983782 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:38.983837 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:38.985620 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:38.986417 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:38.986427 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:38.986538 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:38.987218 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:38.987245 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.988277 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.988281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:38.989690 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:38.989706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:38.990667 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:38.990673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:38.990905 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:38.991778 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:30:38.991781 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:38.991829 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:30:38.991921 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:38.992416 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:38.992434 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:38.992445 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 19:30:38.992680 (+ 235us) client_negotiation.cc:240] Negotiation successful
0923 19:30:38.993179 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":331,"threads_started":1}
I20260923 19:30:38.995633 10171 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46671
I20260923 19:30:38.995916 10171 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:38.996672 10171 heartbeater.cc:511] Master 127.0.97.62:46671 requested a full tablet report, sending...
I20260923 19:30:38.998077 9724 ts_manager.cc:194] Registered new tserver with Master: 2ea500cd929f4b4cb9e422a55c7b2a00 (127.0.97.3:35575)
I20260923 19:30:38.998968 9724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:37037
I20260923 19:30:39.005932 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 19:30:39.012506 10173 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:39.010654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.0.97.62:46671)
0923 19:30:39.010908 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:39.010915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:39.011412 (+ 497us) server_negotiation.cc:407] Connection header received
0923 19:30:39.011569 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:39.011574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:39.011645 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:39.011744 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:39.012245 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56182: BlockingRecv error: recv got EOF from 127.0.0.1:56182 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 19:30:39.013447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9783
I20260923 19:30:39.022773 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:39.014477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34000 (local address 127.0.97.1:35981)
0923 19:30:39.015088 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:39.015099 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:39.015136 (+ 37us) server_negotiation.cc:407] Connection header received
0923 19:30:39.015220 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:39.015231 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:39.015519 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:39.015690 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:39.016215 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:39.017387 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:39.018915 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:39.019621 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:39.019763 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:39.021023 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:39.021033 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:39.021054 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:39.021782 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:39.021840 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:39.021865 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:39.021986 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:39.022172 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 19:30:39.022363 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":242,"threads_started":1}
I20260923 19:30:39.024108 9905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:39.111056 9910 heartbeater.cc:503] Master 127.0.97.62:46671 was elected leader, sending a full tablet report...
I20260923 19:30:39.112354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 9783
W20260923 19:30:39.128393 9708 connection.cc:535] server connection from 127.0.97.1:51739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 19:30:39.128732 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9914
I20260923 19:30:39.137006 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:39.129755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.0.97.2:39939)
0923 19:30:39.130356 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:39.130363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:39.130395 (+ 32us) server_negotiation.cc:407] Connection header received
0923 19:30:39.130464 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:39.130470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:39.130681 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:39.130822 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:39.131276 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:39.132290 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:39.133707 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:39.134361 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:39.134485 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:39.135531 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:39.135539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:39.135553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:39.136207 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:39.136260 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:39.136280 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:39.136389 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:39.136564 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 19:30:39.136676 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":296,"threads_started":1}
I20260923 19:30:39.138163 10036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:39.226028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 9914
I20260923 19:30:39.239882 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 10044
I20260923 19:30:39.248046 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:39.240914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.0.97.3:35575)
0923 19:30:39.241562 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:39.241570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:39.241601 (+ 31us) server_negotiation.cc:407] Connection header received
0923 19:30:39.241665 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:39.241672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:39.241892 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:39.242019 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:39.242589 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:39.243601 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:39.244926 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:39.245475 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:39.245569 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:39.246716 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:39.246725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:39.246739 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:39.247342 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:39.247391 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:39.247410 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:39.247520 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:39.247637 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 19:30:39.247746 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":305,"threads_started":1}
I20260923 19:30:39.249231 10166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:39.336668 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 10044
I20260923 19:30:39.352097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 9693
I20260923 19:30:39.353726 9754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:39.454980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 9693
2026-09-23T19:30:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2666 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T19:30:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:30:39Z Disabled control of system clock
I20260923 19:30:39.531502 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33471
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 19:30:39.868291 10191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:39.868566 10191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:39.868645 10191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:39.875710 10191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:30:39.875813 10191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:39.875849 10191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:39.875895 10191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:30:39.875941 10191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:30:39.884924 10191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:39.886117 10191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:39.887780 10191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:39.899487 10196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:39.899497 10199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:39.899641 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:39.901114 10191 server_base.cc:1034] running on GCE node
I20260923 19:30:39.901860 10191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:39.903357 10191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:39.904567 10191 hybrid_clock.cc:648] HybridClock initialized: now 1790191839904509 us; error 54 us; skew 500 ppm
I20260923 19:30:39.904903 10191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:30:39.907264 10191 webserver.cc:460] Webserver started at http://127.0.97.62:41053/ using document root <none> and password file <none>
I20260923 19:30:39.907930 10191 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:39.908038 10191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:39.908357 10191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:39.911427 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "df41745cba514d6eaf243f5118af048d"
format_stamp: "Formatted at 2026-09-23 19:30:39 on dist-test-slave-kxfb"
server_key: "2e662d957c885d647a4b3d115d9662d7"
server_key_iv: "2d7e2c6d8b6110b61bc76c83eccc3c63"
server_key_version: "encryptionkey@0"
I20260923 19:30:39.912221 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "df41745cba514d6eaf243f5118af048d"
format_stamp: "Formatted at 2026-09-23 19:30:39 on dist-test-slave-kxfb"
server_key: "2e662d957c885d647a4b3d115d9662d7"
server_key_iv: "2d7e2c6d8b6110b61bc76c83eccc3c63"
server_key_version: "encryptionkey@0"
I20260923 19:30:39.917533 10191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:30:39.921116 10205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:39.922642 10191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:30:39.922796 10191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "df41745cba514d6eaf243f5118af048d"
format_stamp: "Formatted at 2026-09-23 19:30:39 on dist-test-slave-kxfb"
server_key: "2e662d957c885d647a4b3d115d9662d7"
server_key_iv: "2d7e2c6d8b6110b61bc76c83eccc3c63"
server_key_version: "encryptionkey@0"
I20260923 19:30:39.922989 10191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:39.928608 10191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:39.929306 10191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:39.929579 10191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:39.947542 10191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33471
I20260923 19:30:39.947608 10256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33471 every 8 connection(s)
I20260923 19:30:39.948794 10191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:30:39.953219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 10191
I20260923 19:30:39.953362 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:39.953438 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:30:39.954276 388 external_mini_cluster.cc:1428] Setting key 044c07bf56a2774e5061173b77bc48fd
I20260923 19:30:39.964061 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:39.956686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.0.97.62:33471)
0923 19:30:39.957830 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:39.957843 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:39.957888 (+ 45us) server_negotiation.cc:407] Connection header received
0923 19:30:39.960366 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:39.960390 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:39.960888 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:39.961267 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:39.962429 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:39.962436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:39.962450 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:39.962465 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:39.962524 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:39.962540 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:39.962752 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:39.962884 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 19:30:39.963134 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":380,"threads_started":1}
I20260923 19:30:39.968601 10257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d: Bootstrap starting.
I20260923 19:30:39.972321 10257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:39.973461 10257 log.cc:824] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:39.976632 10257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d: No bootstrap required, opened a new log
I20260923 19:30:39.984390 10257 raft_consensus.cc:348] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df41745cba514d6eaf243f5118af048d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33471 } }
I20260923 19:30:39.984827 10257 raft_consensus.cc:374] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:39.984925 10257 raft_consensus.cc:729] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df41745cba514d6eaf243f5118af048d, State: Initialized, Role: FOLLOWER
I20260923 19:30:39.985549 10257 consensus_queue.cc:260] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df41745cba514d6eaf243f5118af048d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33471 } }
I20260923 19:30:39.985836 10257 raft_consensus.cc:388] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:30:39.985993 10257 raft_consensus.cc:482] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:30:39.986176 10257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:39.988569 10257 raft_consensus.cc:504] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df41745cba514d6eaf243f5118af048d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33471 } }
I20260923 19:30:39.989208 10257 leader_election.cc:302] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df41745cba514d6eaf243f5118af048d; no voters:
I20260923 19:30:39.989954 10257 leader_election.cc:288] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:30:39.990185 10262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:39.991148 10262 raft_consensus.cc:686] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [term 1 LEADER]: Becoming Leader. State: Replica: df41745cba514d6eaf243f5118af048d, State: Running, Role: LEADER
I20260923 19:30:39.991751 10262 consensus_queue.cc:237] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df41745cba514d6eaf243f5118af048d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33471 } }
I20260923 19:30:39.992408 10257 sys_catalog.cc:564] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:30:39.994563 10263 sys_catalog.cc:455] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df41745cba514d6eaf243f5118af048d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df41745cba514d6eaf243f5118af048d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33471 } } }
I20260923 19:30:39.994980 10263 sys_catalog.cc:458] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:39.995851 10267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:30:39.995671 10264 sys_catalog.cc:455] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df41745cba514d6eaf243f5118af048d. Latest consensus state: current_term: 1 leader_uuid: "df41745cba514d6eaf243f5118af048d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df41745cba514d6eaf243f5118af048d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33471 } } }
I20260923 19:30:39.996984 10264 sys_catalog.cc:458] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:40.001500 10267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:30:40.015733 10267 catalog_manager.cc:1347] Generated new cluster ID: fd8a34b513dc4270bbcc6d91cd060f3e
I20260923 19:30:40.015892 10267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:30:40.028452 10267 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:30:40.028956 10267 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:30:40.038756 10267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df41745cba514d6eaf243f5118af048d: Generated new TSK 0
I20260923 19:30:40.039980 10267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:30:40.049985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33471
--builtin_ntp_servers=127.0.97.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 19:30:40.447428 10281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:40.447809 10281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:40.447921 10281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:40.455904 10281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:40.456053 10281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:40.456219 10281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:30:40.466473 10281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:40.468160 10281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:40.470250 10281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:40.484964 10286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:40.484961 10287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:40.485713 10289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:40.485977 10281 server_base.cc:1034] running on GCE node
I20260923 19:30:40.486711 10281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:40.487769 10281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:40.488996 10281 hybrid_clock.cc:648] HybridClock initialized: now 1790191840488907 us; error 85 us; skew 500 ppm
I20260923 19:30:40.489331 10281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:30:40.491807 10281 webserver.cc:460] Webserver started at http://127.0.97.1:37679/ using document root <none> and password file <none>
I20260923 19:30:40.492538 10281 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:40.492633 10281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:40.492969 10281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:40.497687 10281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "069a914d84fe4bb694c1e86dc89a2437"
format_stamp: "Formatted at 2026-09-23 19:30:40 on dist-test-slave-kxfb"
server_key: "4453e6219d46d484f2de62ceaa7fba4a"
server_key_iv: "deef9eea5ca611b9fea8276a307beaad"
server_key_version: "encryptionkey@0"
I20260923 19:30:40.498507 10281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "069a914d84fe4bb694c1e86dc89a2437"
format_stamp: "Formatted at 2026-09-23 19:30:40 on dist-test-slave-kxfb"
server_key: "4453e6219d46d484f2de62ceaa7fba4a"
server_key_iv: "deef9eea5ca611b9fea8276a307beaad"
server_key_version: "encryptionkey@0"
I20260923 19:30:40.504060 10281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 19:30:40.507972 10295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:40.509640 10281 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:30:40.509812 10281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "069a914d84fe4bb694c1e86dc89a2437"
format_stamp: "Formatted at 2026-09-23 19:30:40 on dist-test-slave-kxfb"
server_key: "4453e6219d46d484f2de62ceaa7fba4a"
server_key_iv: "deef9eea5ca611b9fea8276a307beaad"
server_key_version: "encryptionkey@0"
I20260923 19:30:40.510049 10281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:40.515650 10281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:40.516415 10281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:40.516644 10281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:40.517580 10281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:40.519106 10281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:40.519218 10281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:40.519317 10281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:40.519376 10281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:40.553557 10281 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35137
I20260923 19:30:40.553601 10407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35137 every 8 connection(s)
I20260923 19:30:40.554893 10281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:30:40.560410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 10281
I20260923 19:30:40.560604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:30:40.561275 388 external_mini_cluster.cc:1428] Setting key 6e79cc0bb76cfeaed8f448e480559060
I20260923 19:30:40.564247 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:40.557887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59207 (local address 127.0.97.62:33471)
0923 19:30:40.558406 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:40.558411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:40.559898 (+ 1487us) server_negotiation.cc:407] Connection header received
0923 19:30:40.561701 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:40.561708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:40.561793 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:40.561951 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:40.563401 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:40.563407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:40.563411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:40.563430 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:40.563484 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:40.563499 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:40.563647 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:40.563855 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 19:30:40.563922 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":231,"threads_started":1}
I20260923 19:30:40.564507 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33471
--builtin_ntp_servers=127.0.97.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 19:30:40.565153 10411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:40.558565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33471 (local address 127.0.97.1:59207)
0923 19:30:40.559709 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:40.559751 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:40.561407 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:40.562225 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:40.562236 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:40.562389 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:40.563028 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:30:40.563030 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:40.563084 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:30:40.563213 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:40.563630 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:40.563640 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:40.563647 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 19:30:40.563908 (+ 261us) client_negotiation.cc:240] Negotiation successful
0923 19:30:40.564118 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":343,"threads_started":1}
I20260923 19:30:40.566756 10408 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33471
I20260923 19:30:40.567165 10408 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:40.567965 10408 heartbeater.cc:511] Master 127.0.97.62:33471 requested a full tablet report, sending...
I20260923 19:30:40.570434 10222 ts_manager.cc:194] Registered new tserver with Master: 069a914d84fe4bb694c1e86dc89a2437 (127.0.97.1:35137)
W20260923 19:30:40.924695 10412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:40.925004 10412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:40.925112 10412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:40.932278 10412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:40.932380 10412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:40.932523 10412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:30:40.941545 10412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:40.942896 10412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:40.944741 10412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:40.958751 10417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:40.958948 10418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:40.960402 10412 server_base.cc:1034] running on GCE node
W20260923 19:30:40.960480 10420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:40.961203 10412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:40.962188 10412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:40.963392 10412 hybrid_clock.cc:648] HybridClock initialized: now 1790191840963320 us; error 69 us; skew 500 ppm
I20260923 19:30:40.963713 10412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:30:40.966044 10412 webserver.cc:460] Webserver started at http://127.0.97.2:34299/ using document root <none> and password file <none>
I20260923 19:30:40.966755 10412 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:40.966851 10412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:40.967159 10412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:40.969888 10412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "f82b24ffc969451e8f4a8107c351755d"
format_stamp: "Formatted at 2026-09-23 19:30:40 on dist-test-slave-kxfb"
server_key: "e8ec62306f6a0a92a7e6df7223e55ec3"
server_key_iv: "9502bb853c5bcfc32ae2e5f19d2bafc9"
server_key_version: "encryptionkey@0"
I20260923 19:30:40.970670 10412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f82b24ffc969451e8f4a8107c351755d"
format_stamp: "Formatted at 2026-09-23 19:30:40 on dist-test-slave-kxfb"
server_key: "e8ec62306f6a0a92a7e6df7223e55ec3"
server_key_iv: "9502bb853c5bcfc32ae2e5f19d2bafc9"
server_key_version: "encryptionkey@0"
I20260923 19:30:40.975844 10412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 19:30:40.979524 10426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:40.980971 10412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 19:30:40.981122 10412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "f82b24ffc969451e8f4a8107c351755d"
format_stamp: "Formatted at 2026-09-23 19:30:40 on dist-test-slave-kxfb"
server_key: "e8ec62306f6a0a92a7e6df7223e55ec3"
server_key_iv: "9502bb853c5bcfc32ae2e5f19d2bafc9"
server_key_version: "encryptionkey@0"
I20260923 19:30:40.981329 10412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:40.986874 10412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:40.987660 10412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:40.987901 10412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:40.988848 10412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:40.990247 10412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:40.990363 10412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:40.990468 10412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:40.990537 10412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:41.024312 10412 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41689
I20260923 19:30:41.024394 10538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41689 every 8 connection(s)
I20260923 19:30:41.025667 10412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:30:41.033071 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 10412
I20260923 19:30:41.033223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:30:41.033648 388 external_mini_cluster.cc:1428] Setting key c2c6481a454020b88dccf55809cf74e9
I20260923 19:30:41.035425 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:41.028769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53669 (local address 127.0.97.62:33471)
0923 19:30:41.028989 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:41.028993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:41.030732 (+ 1739us) server_negotiation.cc:407] Connection header received
0923 19:30:41.032566 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:41.032572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:41.032643 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:41.032775 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:41.034529 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:41.034537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:41.034543 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:41.034569 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:41.034614 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:41.034630 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:41.034781 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:41.034996 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 19:30:41.035068 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 19:30:41.036288 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:41.029335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33471 (local address 127.0.97.2:53669)
0923 19:30:41.030549 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:41.030598 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:41.032181 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:41.033078 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:41.033089 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:41.033242 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:41.034078 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:30:41.034081 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:41.034138 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:30:41.034274 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:41.034760 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:41.034771 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:41.034781 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 19:30:41.035047 (+ 266us) client_negotiation.cc:240] Negotiation successful
0923 19:30:41.035277 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":343,"threads_started":1}
I20260923 19:30:41.037122 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33471
--builtin_ntp_servers=127.0.97.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 19:30:41.038043 10539 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33471
I20260923 19:30:41.038380 10539 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:41.039006 10539 heartbeater.cc:511] Master 127.0.97.62:33471 requested a full tablet report, sending...
I20260923 19:30:41.040619 10222 ts_manager.cc:194] Registered new tserver with Master: f82b24ffc969451e8f4a8107c351755d (127.0.97.2:41689)
W20260923 19:30:41.372004 10542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:41.372299 10542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:41.372422 10542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:41.379658 10542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:41.379766 10542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:41.379901 10542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:30:41.389110 10542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:41.390419 10542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:41.392166 10542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:41.405859 10550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:41.406018 10548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:41.406045 10547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:41.407099 10542 server_base.cc:1034] running on GCE node
I20260923 19:30:41.407774 10542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:41.408854 10542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:41.410108 10542 hybrid_clock.cc:648] HybridClock initialized: now 1790191841410023 us; error 76 us; skew 500 ppm
I20260923 19:30:41.410544 10542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:30:41.413023 10542 webserver.cc:460] Webserver started at http://127.0.97.3:45331/ using document root <none> and password file <none>
I20260923 19:30:41.413762 10542 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:41.413864 10542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:41.414166 10542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:41.417130 10542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "d60faa2ce2c8495e90d60206b6e42602"
format_stamp: "Formatted at 2026-09-23 19:30:41 on dist-test-slave-kxfb"
server_key: "5ed300ac3ccc4dc32fc0173d31bf0801"
server_key_iv: "f7aa0816a714881dde6a67d2056874d8"
server_key_version: "encryptionkey@0"
I20260923 19:30:41.417887 10542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d60faa2ce2c8495e90d60206b6e42602"
format_stamp: "Formatted at 2026-09-23 19:30:41 on dist-test-slave-kxfb"
server_key: "5ed300ac3ccc4dc32fc0173d31bf0801"
server_key_iv: "f7aa0816a714881dde6a67d2056874d8"
server_key_version: "encryptionkey@0"
I20260923 19:30:41.423434 10542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 19:30:41.427201 10556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:41.428658 10542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:30:41.428809 10542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "d60faa2ce2c8495e90d60206b6e42602"
format_stamp: "Formatted at 2026-09-23 19:30:41 on dist-test-slave-kxfb"
server_key: "5ed300ac3ccc4dc32fc0173d31bf0801"
server_key_iv: "f7aa0816a714881dde6a67d2056874d8"
server_key_version: "encryptionkey@0"
I20260923 19:30:41.429015 10542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:41.434581 10542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:41.435387 10542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:41.435629 10542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:41.436532 10542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:41.437897 10542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:41.437994 10542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:41.438100 10542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:41.438169 10542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:41.471463 10542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46347
I20260923 19:30:41.471546 10668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46347 every 8 connection(s)
I20260923 19:30:41.472687 10542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:30:41.473268 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 10542
I20260923 19:30:41.473434 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:30:41.474031 388 external_mini_cluster.cc:1428] Setting key 74f92a8616e667e905ea3d171b95222b
I20260923 19:30:41.484898 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:41.475225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44931 (local address 127.0.97.62:33471)
0923 19:30:41.475452 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:41.475459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:41.477329 (+ 1870us) server_negotiation.cc:407] Connection header received
0923 19:30:41.479434 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:41.479438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:41.479492 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:41.479580 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:41.484115 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:41.484121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:41.484125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:41.484142 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:41.484180 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:41.484193 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:41.484304 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:41.484561 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 19:30:41.484619 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 19:30:41.486223 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:41.475790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33471 (local address 127.0.97.3:44931)
0923 19:30:41.477134 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:41.477200 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:41.479155 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:41.479824 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:41.479834 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:41.479947 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:41.483719 (+ 3772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:30:41.483722 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:41.483777 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:30:41.483903 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:41.484296 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:41.484302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:41.484309 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 19:30:41.484618 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 19:30:41.484970 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":356,"threads_started":1}
I20260923 19:30:41.487844 10669 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33471
I20260923 19:30:41.488207 10669 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:41.489054 10669 heartbeater.cc:511] Master 127.0.97.62:33471 requested a full tablet report, sending...
I20260923 19:30:41.490799 10222 ts_manager.cc:194] Registered new tserver with Master: d60faa2ce2c8495e90d60206b6e42602 (127.0.97.3:46347)
I20260923 19:30:41.503590 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 19:30:41.510052 10410 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:30:41.508346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.0.97.62:33471)
0923 19:30:41.508606 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:41.508614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:41.509081 (+ 467us) server_negotiation.cc:407] Connection header received
0923 19:30:41.509269 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:41.509273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:41.509329 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:41.509422 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:41.509749 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52798: BlockingRecv error: recv got EOF from 127.0.0.1:52798 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 19:30:41.511217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 10281
I20260923 19:30:41.515368 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:41.512217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.0.97.1:35137)
0923 19:30:41.512797 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:41.512805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:41.512836 (+ 31us) server_negotiation.cc:407] Connection header received
0923 19:30:41.512909 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:41.512917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:41.513208 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:41.513356 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:41.514476 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:41.514484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:41.514500 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:41.514527 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:41.514601 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:41.514625 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:41.514758 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:41.514925 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 19:30:41.515021 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":289,"threads_started":1}
I20260923 19:30:41.516762 10403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:41.603433 10408 heartbeater.cc:503] Master 127.0.97.62:33471 was elected leader, sending a full tablet report...
I20260923 19:30:41.604986 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 10281
I20260923 19:30:41.619376 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 10412
I20260923 19:30:41.623071 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:41.620340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54596 (local address 127.0.97.2:41689)
0923 19:30:41.620923 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:41.620932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:41.620965 (+ 33us) server_negotiation.cc:407] Connection header received
0923 19:30:41.621041 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:41.621049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:41.621340 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:41.621493 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:41.622458 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:41.622464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:41.622477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:41.622488 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:41.622536 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:41.622550 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:41.622666 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:41.622715 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 19:30:41.622807 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":295,"threads_started":1}
I20260923 19:30:41.624433 10534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:41.706393 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 10412
I20260923 19:30:41.719702 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 10542
I20260923 19:30:41.724120 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:41.720720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.97.3:46347)
0923 19:30:41.721557 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:41.721567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:41.721605 (+ 38us) server_negotiation.cc:407] Connection header received
0923 19:30:41.721692 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:41.721701 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:41.721992 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:41.722149 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:41.723364 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:41.723371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:41.723385 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:30:41.723402 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:41.723446 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:30:41.723460 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:41.723601 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:41.723665 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 19:30:41.723773 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":435,"threads_started":1}
I20260923 19:30:41.725643 10664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:41.826398 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 10542
I20260923 19:30:41.842072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 10191
I20260923 19:30:41.843578 10252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:41.943110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 10191
2026-09-23T19:30:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2485 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:30:42 dist-test-slave-kxfb krb5kdc[10687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:30:42 dist-test-slave-kxfb krb5kdc[10687](info): set up 2 sockets
Sep 23 19:30:42 dist-test-slave-kxfb krb5kdc[10687](info): commencing operation
krb5kdc: starting...
W20260923 19:30:42.971342 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:30:43 dist-test-slave-kxfb krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:30:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:30:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:30:43.230602 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41183
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:43.597277 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:43.597630 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:43.597760 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:43.605471 10700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:30:43.605611 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:43.605669 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:43.605751 10700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:30:43.605811 10700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:30:43.616436 10700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:43.618089 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:43.619668 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:43.632699 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:43.632653 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:43.632742 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:43.635870 10700 server_base.cc:1034] running on GCE node
I20260923 19:30:43.636797 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:43.638528 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:43.639755 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790191843639695 us; error 47 us; skew 500 ppm
I20260923 19:30:43.640136 10700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:43 dist-test-slave-kxfb krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:43.648592 10700 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:30:43.650580 10700 webserver.cc:460] Webserver started at http://127.0.97.62:34343/ using document root <none> and password file <none>
I20260923 19:30:43.651417 10700 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:43.651537 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:43.651862 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:43.655310 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040"
format_stamp: "Formatted at 2026-09-23 19:30:43 on dist-test-slave-kxfb"
server_key: "c53c04e201fb6486b1927c1f8b076613"
server_key_iv: "520eeda4959456468313c3dfddcceff2"
server_key_version: "encryptionkey@0"
I20260923 19:30:43.656107 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040"
format_stamp: "Formatted at 2026-09-23 19:30:43 on dist-test-slave-kxfb"
server_key: "c53c04e201fb6486b1927c1f8b076613"
server_key_iv: "520eeda4959456468313c3dfddcceff2"
server_key_version: "encryptionkey@0"
I20260923 19:30:43.661727 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 19:30:43.665488 10715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:43.666966 10700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:30:43.667136 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040"
format_stamp: "Formatted at 2026-09-23 19:30:43 on dist-test-slave-kxfb"
server_key: "c53c04e201fb6486b1927c1f8b076613"
server_key_iv: "520eeda4959456468313c3dfddcceff2"
server_key_version: "encryptionkey@0"
I20260923 19:30:43.667327 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:43.698081 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:43.702847 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:43.703166 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:43.722364 10700 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41183
I20260923 19:30:43.722383 10766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41183 every 8 connection(s)
I20260923 19:30:43.723937 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:30:43.728014 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 10700
I20260923 19:30:43.728216 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:30:43.728507 10767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:43.728935 388 external_mini_cluster.cc:1428] Setting key ef162ec82bd14eac9bb85635a12d4c39
I20260923 19:30:43.742923 10767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040: Bootstrap starting.
Sep 23 19:30:43 dist-test-slave-kxfb krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:30:43.747205 10767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:43.748486 10767 log.cc:824] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:43.751995 10767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040: No bootstrap required, opened a new log
I20260923 19:30:43.752915 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:43.731158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.0.97.62:41183)
0923 19:30:43.732064 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:43.732077 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:43.732121 (+ 44us) server_negotiation.cc:407] Connection header received
0923 19:30:43.733768 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:43.733794 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:43.734261 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:43.734725 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:43.736223 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:43.737595 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:43.738935 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:43.740108 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:43.740301 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:43.744800 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:43.744814 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:43.744832 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:43.744903 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:43.748213 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:43.749029 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:43.749037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:43.749045 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:43.749245 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:43.749627 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:43.749631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:43.749633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:43.750354 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:43.750679 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:43.751131 (+ 452us) server_negotiation.cc:299] Negotiation successful
0923 19:30:43.751608 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":411,"threads_started":1}
I20260923 19:30:43.760624 10767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41183 } }
I20260923 19:30:43.761181 10767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:43.761283 10767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b00d415ab4c43b0ba44dd0ab4ea4040, State: Initialized, Role: FOLLOWER
I20260923 19:30:43.761981 10767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41183 } }
I20260923 19:30:43.762209 10767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:30:43.762336 10767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:30:43.762491 10767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:43.764779 10767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41183 } }
I20260923 19:30:43.765328 10767 leader_election.cc:302] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b00d415ab4c43b0ba44dd0ab4ea4040; no voters:
I20260923 19:30:43.765901 10767 leader_election.cc:288] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:30:43.766048 10772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:43.767320 10772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [term 1 LEADER]: Becoming Leader. State: Replica: 7b00d415ab4c43b0ba44dd0ab4ea4040, State: Running, Role: LEADER
I20260923 19:30:43.767899 10772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41183 } }
I20260923 19:30:43.768258 10767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:30:43.770743 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b00d415ab4c43b0ba44dd0ab4ea4040. Latest consensus state: current_term: 1 leader_uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41183 } } }
I20260923 19:30:43.771163 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:43.771088 10773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b00d415ab4c43b0ba44dd0ab4ea4040" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41183 } } }
I20260923 19:30:43.771420 10773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040 [sys.catalog]: This master's current role is: LEADER
I20260923 19:30:43.774183 10781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:30:43.780947 10781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:30:43.791146 10781 catalog_manager.cc:1347] Generated new cluster ID: f627d5ebfe3947dd806e36ea5472bfb6
I20260923 19:30:43.791285 10781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:30:43.818826 10781 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:30:43.820751 10781 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:30:43.829140 10781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040: Generated new TSK 0
I20260923 19:30:43.830135 10781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:30:43.891850 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41183
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:44.265259 10793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:44.265606 10793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:44.265712 10793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:44.273672 10793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:44.273804 10793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:44.273939 10793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:30:44.284593 10793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:44.286404 10793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:44.287922 10793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:44.302495 10801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:44.302556 10798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:44.302551 10799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:44.303380 10793 server_base.cc:1034] running on GCE node
I20260923 19:30:44.304047 10793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:44.305059 10793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:44.306324 10793 hybrid_clock.cc:648] HybridClock initialized: now 1790191844306239 us; error 46 us; skew 500 ppm
I20260923 19:30:44.306691 10793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:44 dist-test-slave-kxfb krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:44.310078 10793 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:30:44.311856 10793 webserver.cc:460] Webserver started at http://127.0.97.1:46169/ using document root <none> and password file <none>
I20260923 19:30:44.312628 10793 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:44.312731 10793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:44.313010 10793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:44.316241 10793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "e420aef6fcd6454bb1bc283e75dd5612"
format_stamp: "Formatted at 2026-09-23 19:30:44 on dist-test-slave-kxfb"
server_key: "419fb92ce8dcb9d21252199011e7baaa"
server_key_iv: "a46ede1d06705d74d6706d99006c6d61"
server_key_version: "encryptionkey@0"
I20260923 19:30:44.316978 10793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e420aef6fcd6454bb1bc283e75dd5612"
format_stamp: "Formatted at 2026-09-23 19:30:44 on dist-test-slave-kxfb"
server_key: "419fb92ce8dcb9d21252199011e7baaa"
server_key_iv: "a46ede1d06705d74d6706d99006c6d61"
server_key_version: "encryptionkey@0"
I20260923 19:30:44.322911 10793 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260923 19:30:44.326881 10808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:44.328717 10793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:30:44.328897 10793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "e420aef6fcd6454bb1bc283e75dd5612"
format_stamp: "Formatted at 2026-09-23 19:30:44 on dist-test-slave-kxfb"
server_key: "419fb92ce8dcb9d21252199011e7baaa"
server_key_iv: "a46ede1d06705d74d6706d99006c6d61"
server_key_version: "encryptionkey@0"
I20260923 19:30:44.329109 10793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:44.360144 10793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:44.364886 10793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:44.365202 10793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:44.366312 10793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:44.367811 10793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:44.367890 10793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:44.368002 10793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:44.368053 10793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:44.402719 10793 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45255
I20260923 19:30:44.402812 10920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45255 every 8 connection(s)
I20260923 19:30:44.404191 10793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:30:44.410108 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 10793
I20260923 19:30:44.410351 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:30:44.411129 388 external_mini_cluster.cc:1428] Setting key 6bb59306c2f693f8387833ba3bcd9080
Sep 23 19:30:44 dist-test-slave-kxfb krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:30:44.426931 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:44.407173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48171 (local address 127.0.97.62:41183)
0923 19:30:44.407659 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:44.407664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:44.408705 (+ 1041us) server_negotiation.cc:407] Connection header received
0923 19:30:44.411194 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:44.411203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:44.411338 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:44.411464 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:44.413097 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:44.414145 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:44.415907 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:44.416425 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:44.416545 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:44.420566 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:44.420573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:44.420579 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:44.420631 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:44.423830 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:44.424576 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:44.424581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:44.424585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:44.424735 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:44.425131 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:44.425135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:44.425138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:44.425457 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:44.425606 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:44.426276 (+ 670us) server_negotiation.cc:299] Negotiation successful
0923 19:30:44.426552 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":266,"threads_started":1}
I20260923 19:30:44.427735 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:44.407728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41183 (local address 127.0.97.1:48171)
0923 19:30:44.408532 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:44.408579 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:44.410769 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:44.411758 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:44.411767 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:44.412299 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:44.412901 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:44.412917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:44.414347 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:44.414352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:44.415746 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:44.415763 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:44.416630 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:44.416635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:44.416873 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:44.417594 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:44.417619 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:44.420288 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:44.423989 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:44.423995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:44.424008 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:44.424450 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:44.424827 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:44.424829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:44.424832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:44.425028 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:44.425570 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:44.425576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:44.425907 (+ 331us) client_negotiation.cc:769] Sending connection context
0923 19:30:44.426129 (+ 222us) client_negotiation.cc:240] Negotiation successful
0923 19:30:44.426590 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":327,"threads_started":1}
I20260923 19:30:44.429454 10921 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41183
I20260923 19:30:44.429740 10921 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:44.430495 10921 heartbeater.cc:511] Master 127.0.97.62:41183 requested a full tablet report, sending...
I20260923 19:30:44.432942 10732 ts_manager.cc:194] Registered new tserver with Master: e420aef6fcd6454bb1bc283e75dd5612 (127.0.97.1:45255)
I20260923 19:30:44.435276 10732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48171
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:30:44.469278 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41183
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:44.834417 10927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:44.834757 10927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:44.834887 10927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:44.842667 10927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:44.842804 10927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:44.842957 10927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:30:44.853616 10927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:44.855384 10927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:44.856961 10927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:44.872634 10935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:44.872593 10933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:44.872613 10932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:44.874099 10927 server_base.cc:1034] running on GCE node
I20260923 19:30:44.874876 10927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:44.876019 10927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:44.877270 10927 hybrid_clock.cc:648] HybridClock initialized: now 1790191844877202 us; error 51 us; skew 500 ppm
I20260923 19:30:44.877627 10927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:44 dist-test-slave-kxfb krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:44.880975 10927 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:30:44.882617 10927 webserver.cc:460] Webserver started at http://127.0.97.2:42377/ using document root <none> and password file <none>
I20260923 19:30:44.883460 10927 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:44.883582 10927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:44.883899 10927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:44.887298 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf"
format_stamp: "Formatted at 2026-09-23 19:30:44 on dist-test-slave-kxfb"
server_key: "095e6efd5d35149595ad35e2eece4ba3"
server_key_iv: "90edaee781454d1776a88f8d7125e069"
server_key_version: "encryptionkey@0"
I20260923 19:30:44.888108 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf"
format_stamp: "Formatted at 2026-09-23 19:30:44 on dist-test-slave-kxfb"
server_key: "095e6efd5d35149595ad35e2eece4ba3"
server_key_iv: "90edaee781454d1776a88f8d7125e069"
server_key_version: "encryptionkey@0"
I20260923 19:30:44.893848 10927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:30:44.897882 10942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:44.899477 10927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:30:44.899632 10927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf"
format_stamp: "Formatted at 2026-09-23 19:30:44 on dist-test-slave-kxfb"
server_key: "095e6efd5d35149595ad35e2eece4ba3"
server_key_iv: "90edaee781454d1776a88f8d7125e069"
server_key_version: "encryptionkey@0"
I20260923 19:30:44.899845 10927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:44.922813 10927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:44.926817 10927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:44.927052 10927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:44.928133 10927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:44.929544 10927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:44.929642 10927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:44.929768 10927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:44.929822 10927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:44.962819 10927 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38201
I20260923 19:30:44.962932 11054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38201 every 8 connection(s)
I20260923 19:30:44.964159 10927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:30:44.964737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 10927
I20260923 19:30:44.964937 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:30:44.965502 388 external_mini_cluster.cc:1428] Setting key 237444d7771f3ebfbf871fc8c4e46189
Sep 23 19:30:44 dist-test-slave-kxfb krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:30:44.986917 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:44.967333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53041 (local address 127.0.97.62:41183)
0923 19:30:44.967884 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:44.967891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:44.969294 (+ 1403us) server_negotiation.cc:407] Connection header received
0923 19:30:44.971742 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:44.971746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:44.971840 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:44.972000 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:44.973824 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:44.974706 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:44.976306 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:44.976733 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:44.976839 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:44.981203 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:44.981208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:44.981212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:44.981248 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:44.983832 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:44.984586 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:44.984592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:44.984595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:44.984731 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:44.985130 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:44.985135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:44.985138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:44.985419 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:44.985566 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:44.986265 (+ 699us) server_negotiation.cc:299] Negotiation successful
0923 19:30:44.986515 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":297,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:30:44.988086 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:44.968023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41183 (local address 127.0.97.2:53041)
0923 19:30:44.969078 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:44.969147 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:44.971405 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:44.972276 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:44.972287 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:44.972937 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:44.973629 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:44.973659 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:44.974856 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:44.974861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:44.976174 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:44.976187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:44.976950 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:44.976957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:44.977185 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:44.977880 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:44.977903 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:44.980912 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:44.983987 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:44.983998 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:44.984019 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:44.984476 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:44.984831 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:44.984836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:44.984840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:44.985031 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:44.985537 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:44.985545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:44.985916 (+ 371us) client_negotiation.cc:769] Sending connection context
0923 19:30:44.986186 (+ 270us) client_negotiation.cc:240] Negotiation successful
0923 19:30:44.986812 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":370,"threads_started":1}
I20260923 19:30:44.989852 11055 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 19:30:44.990228 11055 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:44.991073 11055 heartbeater.cc:511] Master 127.0.97.62:41183 requested a full tablet report, sending...
I20260923 19:30:44.992779 10732 ts_manager.cc:194] Registered new tserver with Master: a4bf4cebb2624b8a99b8bee1f7807ecf (127.0.97.2:38201)
I20260923 19:30:44.993664 10732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:30:45.016847 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41183
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:30:45.352298 11061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:30:45.352679 11061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:30:45.352828 11061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:30:45.360658 11061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:30:45.360777 11061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:30:45.360918 11061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:30:45.370906 11061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:30:45.372264 11061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:30:45.373566 11061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:30:45.386878 11067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:45.386932 11066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:30:45.386830 11069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:30:45.391845 11061 server_base.cc:1034] running on GCE node
I20260923 19:30:45.392520 11061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:30:45.393498 11061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:30:45.394762 11061 hybrid_clock.cc:648] HybridClock initialized: now 1790191845394699 us; error 50 us; skew 500 ppm
I20260923 19:30:45.395134 11061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:30:45 dist-test-slave-kxfb krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:30:45.398667 11061 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:30:45.400545 11061 webserver.cc:460] Webserver started at http://127.0.97.3:45663/ using document root <none> and password file <none>
I20260923 19:30:45.401432 11061 fs_manager.cc:359] Metadata directory not provided
I20260923 19:30:45.401561 11061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:30:45.401921 11061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:30:45.405480 11061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "581c67bc0f0c44c6b8d417c9ae9c1899"
format_stamp: "Formatted at 2026-09-23 19:30:45 on dist-test-slave-kxfb"
server_key: "628933fef470eb10e866b03dd22eb755"
server_key_iv: "4cb917b65ce3c422129f8ee81715fc6f"
server_key_version: "encryptionkey@0"
I20260923 19:30:45.406347 11061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "581c67bc0f0c44c6b8d417c9ae9c1899"
format_stamp: "Formatted at 2026-09-23 19:30:45 on dist-test-slave-kxfb"
server_key: "628933fef470eb10e866b03dd22eb755"
server_key_iv: "4cb917b65ce3c422129f8ee81715fc6f"
server_key_version: "encryptionkey@0"
I20260923 19:30:45.412302 11061 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 19:30:45.416479 11076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:45.418202 11061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 19:30:45.418427 11061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "581c67bc0f0c44c6b8d417c9ae9c1899"
format_stamp: "Formatted at 2026-09-23 19:30:45 on dist-test-slave-kxfb"
server_key: "628933fef470eb10e866b03dd22eb755"
server_key_iv: "4cb917b65ce3c422129f8ee81715fc6f"
server_key_version: "encryptionkey@0"
I20260923 19:30:45.418677 11061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:30:45.438823 10921 heartbeater.cc:503] Master 127.0.97.62:41183 was elected leader, sending a full tablet report...
I20260923 19:30:45.451445 11061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:30:45.456223 11061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:30:45.456530 11061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:30:45.457669 11061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:30:45.459283 11061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:30:45.459364 11061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:45.459489 11061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:30:45.459543 11061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:30:45.493958 11061 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43909
I20260923 19:30:45.494041 11188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43909 every 8 connection(s)
I20260923 19:30:45.495527 11061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:30:45.499926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 11061
I20260923 19:30:45.500100 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:30:45.500640 388 external_mini_cluster.cc:1428] Setting key 48a319d4de5ac13ac24c9a17f8049d7f
Sep 23 19:30:45 dist-test-slave-kxfb krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:30:45.517784 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.498407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57739 (local address 127.0.97.62:41183)
0923 19:30:45.498816 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.498821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.500295 (+ 1474us) server_negotiation.cc:407] Connection header received
0923 19:30:45.502580 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.502584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.502731 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.502855 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:45.504898 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.505734 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.507600 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.508056 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.508182 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.511654 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:45.511659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:45.511663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:45.511710 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:45.514438 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:45.515154 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:45.515158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:45.515160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:45.515289 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:45.515751 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:45.515755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:45.515758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:45.516104 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:45.516263 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.517096 (+ 833us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.517356 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":223,"threads_started":1}
I20260923 19:30:45.518733 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.499118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41183 (local address 127.0.97.3:57739)
0923 19:30:45.500066 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:45.500129 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:45.502209 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:45.503181 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:45.503192 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:45.503954 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:30:45.504617 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.504634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.505919 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.505923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.507440 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.507454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.508269 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.508274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.508474 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.509124 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:30:45.509146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:30:45.511420 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:30:45.514578 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:45.514586 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:45.514601 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:45.515011 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:45.515403 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:30:45.515406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:30:45.515410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:30:45.515628 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:30:45.516241 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:30:45.516248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:30:45.516650 (+ 402us) client_negotiation.cc:769] Sending connection context
0923 19:30:45.516936 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 19:30:45.517455 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":315,"threads_started":1}
I20260923 19:30:45.520071 11189 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41183
I20260923 19:30:45.520329 11189 heartbeater.cc:464] Registering TS with master...
I20260923 19:30:45.520979 11189 heartbeater.cc:511] Master 127.0.97.62:41183 requested a full tablet report, sending...
I20260923 19:30:45.522395 10732 ts_manager.cc:194] Registered new tserver with Master: 581c67bc0f0c44c6b8d417c9ae9c1899 (127.0.97.3:43909)
I20260923 19:30:45.523319 10732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57739
I20260923 19:30:45.529444 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:30:45.547971 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.534664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.0.97.62:41183)
0923 19:30:45.534884 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.534891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.535458 (+ 567us) server_negotiation.cc:407] Connection header received
0923 19:30:45.535628 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.535632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.535706 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.535795 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:45.537397 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.538132 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.539556 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.540077 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.540204 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.542822 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:45.542828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:45.542831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:45.542900 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:45.545405 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:45.546076 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:45.546080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:45.546082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:45.546200 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:45.546618 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:45.546621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:45.546624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:45.546946 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:45.547135 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.547470 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.547646 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 19:30:45.552574 10732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:30:45.559312 10732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:30:45.585418 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.573032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38201 (local address 127.0.0.1:51348)
0923 19:30:45.575183 (+ 2151us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:45.575228 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:45.575409 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:45.576320 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:45.576330 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:45.576369 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:45.576876 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.576891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.579352 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.579358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.582015 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.582034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.584209 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.584216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.584453 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.584523 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 19:30:45.584669 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 19:30:45.584943 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1894,"thread_start_us":1756,"threads_started":1}
I20260923 19:30:45.588833 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.573304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.97.2:38201)
0923 19:30:45.574504 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.574516 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.575355 (+ 839us) server_negotiation.cc:407] Connection header received
0923 19:30:45.575661 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.575672 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.576015 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.577108 (+ 1093us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:45.577386 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.579177 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.582263 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.583967 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.587950 (+ 3983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.588208 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.588282 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.588478 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":776,"threads_started":1}
I20260923 19:30:45.592944 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.573787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45255 (local address 127.0.0.1:60954)
0923 19:30:45.585755 (+ 11968us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:45.585781 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:45.585922 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:45.587157 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:45.587163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:45.587191 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:45.587381 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.587390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.589242 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.589247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.590856 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.590867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.592153 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.592158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.592341 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.592375 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:30:45.592457 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 19:30:45.592607 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11848,"thread_start_us":13782,"threads_started":1}
I20260923 19:30:45.593012 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.574423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.0.97.1:45255)
0923 19:30:45.575078 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.575089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.586082 (+ 10993us) server_negotiation.cc:407] Connection header received
0923 19:30:45.586226 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.586236 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.586641 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.586874 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:45.587578 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.589071 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.591007 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.591997 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.592119 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.592447 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.592515 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.592683 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":361,"threads_started":1}
I20260923 19:30:45.592571 10990 tablet_service.cc:1467] Processing CreateTablet for tablet dba3678f8b804e3182d4b2673dcd1ce0 (DEFAULT_TABLE table=test-table [id=7fa8a4687d254f9dad1cd151690fd545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:45.593828 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba3678f8b804e3182d4b2673dcd1ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:45.597518 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.575847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43909 (local address 127.0.0.1:42078)
0923 19:30:45.586771 (+ 10924us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:45.586790 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:45.586925 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:45.587525 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:45.587530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:45.587559 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:45.587958 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.587967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.589650 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.589655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.594794 (+ 5139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.594806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.596670 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.596676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.596853 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.596886 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 19:30:45.596977 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 19:30:45.597139 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10792,"thread_start_us":10621,"threads_started":1}
I20260923 19:30:45.597563 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.575984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.0.97.3:43909)
0923 19:30:45.585184 (+ 9200us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.585193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.587032 (+ 1839us) server_negotiation.cc:407] Connection header received
0923 19:30:45.587122 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.587128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.587334 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.587510 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:45.588126 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.589494 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.594976 (+ 5482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.596486 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.596617 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.596876 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.597030 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.597210 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9008,"thread_start_us":9222,"threads_started":1}
I20260923 19:30:45.597108 10856 tablet_service.cc:1467] Processing CreateTablet for tablet dba3678f8b804e3182d4b2673dcd1ce0 (DEFAULT_TABLE table=test-table [id=7fa8a4687d254f9dad1cd151690fd545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:45.598908 10856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba3678f8b804e3182d4b2673dcd1ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:45.601574 11124 tablet_service.cc:1467] Processing CreateTablet for tablet dba3678f8b804e3182d4b2673dcd1ce0 (DEFAULT_TABLE table=test-table [id=7fa8a4687d254f9dad1cd151690fd545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:30:45.603277 11124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba3678f8b804e3182d4b2673dcd1ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:30:45.611258 11208 tablet_bootstrap.cc:492] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: Bootstrap starting.
I20260923 19:30:45.613214 11209 tablet_bootstrap.cc:492] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: Bootstrap starting.
I20260923 19:30:45.616308 11209 tablet_bootstrap.cc:654] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:45.616479 11210 tablet_bootstrap.cc:492] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: Bootstrap starting.
I20260923 19:30:45.617543 11209 log.cc:824] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:45.617616 11208 tablet_bootstrap.cc:654] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:45.618957 11208 log.cc:824] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:45.619778 11210 tablet_bootstrap.cc:654] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: Neither blocks nor log segments found. Creating new log.
I20260923 19:30:45.620977 11209 tablet_bootstrap.cc:492] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: No bootstrap required, opened a new log
I20260923 19:30:45.620944 11210 log.cc:824] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: Log is configured to *not* fsync() on all Append() calls
I20260923 19:30:45.621237 11209 ts_tablet_manager.cc:1397] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260923 19:30:45.621685 11208 tablet_bootstrap.cc:492] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: No bootstrap required, opened a new log
I20260923 19:30:45.621941 11208 ts_tablet_manager.cc:1397] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260923 19:30:45.630254 11210 tablet_bootstrap.cc:492] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: No bootstrap required, opened a new log
I20260923 19:30:45.630568 11210 ts_tablet_manager.cc:1397] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260923 19:30:45.630522 11208 raft_consensus.cc:348] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } }
I20260923 19:30:45.630973 11208 raft_consensus.cc:374] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:45.631156 11208 raft_consensus.cc:729] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bf4cebb2624b8a99b8bee1f7807ecf, State: Initialized, Role: FOLLOWER
I20260923 19:30:45.631829 11208 consensus_queue.cc:260] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } }
I20260923 19:30:45.631922 11209 raft_consensus.cc:348] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } }
I20260923 19:30:45.632372 11209 raft_consensus.cc:374] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:45.632472 11209 raft_consensus.cc:729] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e420aef6fcd6454bb1bc283e75dd5612, State: Initialized, Role: FOLLOWER
I20260923 19:30:45.633112 11209 consensus_queue.cc:260] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } }
I20260923 19:30:45.634984 11209 ts_tablet_manager.cc:1428] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260923 19:30:45.641701 11210 raft_consensus.cc:348] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } }
I20260923 19:30:45.642165 11210 raft_consensus.cc:374] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:30:45.642256 11210 raft_consensus.cc:729] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581c67bc0f0c44c6b8d417c9ae9c1899, State: Initialized, Role: FOLLOWER
I20260923 19:30:45.642670 11055 heartbeater.cc:503] Master 127.0.97.62:41183 was elected leader, sending a full tablet report...
I20260923 19:30:45.643028 11210 consensus_queue.cc:260] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } }
I20260923 19:30:45.643577 11208 ts_tablet_manager.cc:1428] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20260923 19:30:45.644627 11189 heartbeater.cc:503] Master 127.0.97.62:41183 was elected leader, sending a full tablet report...
I20260923 19:30:45.645228 11210 ts_tablet_manager.cc:1428] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260923 19:30:45.656718 10922 tablet.cc:2369] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:30:45.692339 11214 raft_consensus.cc:482] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:30:45.692643 11214 raft_consensus.cc:504] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } }
I20260923 19:30:45.694420 11214 leader_election.cc:288] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 581c67bc0f0c44c6b8d417c9ae9c1899 (127.0.97.3:43909), e420aef6fcd6454bb1bc283e75dd5612 (127.0.97.1:45255)
I20260923 19:30:45.702280 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.694869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46631 (local address 127.0.97.1:45255)
0923 19:30:45.695068 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.695074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.696546 (+ 1472us) server_negotiation.cc:407] Connection header received
0923 19:30:45.696841 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.696847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.696926 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.697026 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:45.697631 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.698500 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.700473 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.701228 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.701336 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.701448 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.701890 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.702023 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 19:30:45.702474 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.694998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54281 (local address 127.0.97.3:43909)
0923 19:30:45.695199 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.695204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.696043 (+ 839us) server_negotiation.cc:407] Connection header received
0923 19:30:45.696254 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.696260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.696335 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.696428 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:30:45.697160 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.698199 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.700483 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.701302 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.701443 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.701566 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.702023 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.702174 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 19:30:45.702586 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.695343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45255 (local address 127.0.97.2:46631)
0923 19:30:45.696392 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:45.696415 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:45.696656 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:45.697054 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:45.697058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:45.697089 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:45.697461 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.697472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.698641 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.698647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.700321 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.700333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.701471 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.701477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.701659 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.701761 (+ 102us) client_negotiation.cc:769] Sending connection context
0923 19:30:45.701837 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 19:30:45.701954 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":49,"thread_start_us":665,"threads_started":1}
I20260923 19:30:45.702653 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.694984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43909 (local address 127.0.97.2:54281)
0923 19:30:45.695905 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0923 19:30:45.695924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:30:45.696109 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:30:45.696541 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:30:45.696547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:30:45.696593 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:30:45.696974 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.696984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.698365 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.698372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.700304 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:30:45.700319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.701554 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.701559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:30:45.701724 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.701819 (+ 95us) client_negotiation.cc:769] Sending connection context
0923 19:30:45.701978 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 19:30:45.702139 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":98,"thread_start_us":514,"threads_started":1}
I20260923 19:30:45.703817 11144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba3678f8b804e3182d4b2673dcd1ce0" candidate_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" is_pre_election: true
I20260923 19:30:45.703816 10876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba3678f8b804e3182d4b2673dcd1ce0" candidate_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e420aef6fcd6454bb1bc283e75dd5612" is_pre_election: true
I20260923 19:30:45.704412 11144 raft_consensus.cc:2454] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf4cebb2624b8a99b8bee1f7807ecf in term 0.
I20260923 19:30:45.704412 10876 raft_consensus.cc:2454] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bf4cebb2624b8a99b8bee1f7807ecf in term 0.
I20260923 19:30:45.705288 10943 leader_election.cc:302] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf4cebb2624b8a99b8bee1f7807ecf, e420aef6fcd6454bb1bc283e75dd5612; no voters:
I20260923 19:30:45.705765 11214 raft_consensus.cc:2781] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:30:45.705925 11214 raft_consensus.cc:482] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:30:45.706032 11214 raft_consensus.cc:3037] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:45.708263 11214 raft_consensus.cc:504] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } }
I20260923 19:30:45.708931 11214 leader_election.cc:288] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [CANDIDATE]: Term 1 election: Requested vote from peers 581c67bc0f0c44c6b8d417c9ae9c1899 (127.0.97.3:43909), e420aef6fcd6454bb1bc283e75dd5612 (127.0.97.1:45255)
I20260923 19:30:45.709568 10876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba3678f8b804e3182d4b2673dcd1ce0" candidate_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e420aef6fcd6454bb1bc283e75dd5612"
I20260923 19:30:45.709551 11144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba3678f8b804e3182d4b2673dcd1ce0" candidate_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899"
I20260923 19:30:45.709756 10876 raft_consensus.cc:3037] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:45.709824 11144 raft_consensus.cc:3037] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:30:45.711923 10876 raft_consensus.cc:2454] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf4cebb2624b8a99b8bee1f7807ecf in term 1.
I20260923 19:30:45.712374 10943 leader_election.cc:302] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bf4cebb2624b8a99b8bee1f7807ecf, e420aef6fcd6454bb1bc283e75dd5612; no voters:
I20260923 19:30:45.712500 11144 raft_consensus.cc:2454] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bf4cebb2624b8a99b8bee1f7807ecf in term 1.
I20260923 19:30:45.712714 11214 raft_consensus.cc:2781] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:30:45.713210 11214 raft_consensus.cc:686] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 1 LEADER]: Becoming Leader. State: Replica: a4bf4cebb2624b8a99b8bee1f7807ecf, State: Running, Role: LEADER
I20260923 19:30:45.713758 11214 consensus_queue.cc:237] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } }
W20260923 19:30:45.716547 11056 tablet.cc:2369] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:30:45.718245 10731 catalog_manager.cc:5495] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bf4cebb2624b8a99b8bee1f7807ecf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bf4cebb2624b8a99b8bee1f7807ecf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 } health_report { overall_health: UNKNOWN } } }
I20260923 19:30:45.741030 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.735707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.0.97.2:38201)
0923 19:30:45.735895 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.735901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.736052 (+ 151us) server_negotiation.cc:407] Connection header received
0923 19:30:45.736283 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.736287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.736373 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.736505 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:30:45.736884 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.737583 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.738959 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.739342 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.739451 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.739921 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:30:45.740445 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:30:45.740568 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.740674 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.740758 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 19:30:45.747273 11190 tablet.cc:2369] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:30:45.750412 10876 raft_consensus.cc:1261] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [term 1 FOLLOWER]: Refusing update from remote peer a4bf4cebb2624b8a99b8bee1f7807ecf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:45.750412 11144 raft_consensus.cc:1261] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [term 1 FOLLOWER]: Refusing update from remote peer a4bf4cebb2624b8a99b8bee1f7807ecf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:30:45.751525 11219 consensus_queue.cc:1035] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "581c67bc0f0c44c6b8d417c9ae9c1899" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:45.751929 11214 consensus_queue.cc:1035] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e420aef6fcd6454bb1bc283e75dd5612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:30:45.764566 11224 mvcc.cc:205] Tried to move back new op lower bound from 7332625800182693888 to 7332625800047570944. Current Snapshot: MvccSnapshot[applied={T|T < 7332625800182693888}]
I20260923 19:30:45.765626 11222 mvcc.cc:205] Tried to move back new op lower bound from 7332625800182693888 to 7332625800047570944. Current Snapshot: MvccSnapshot[applied={T|T < 7332625800182693888}]
I20260923 19:30:45.766505 11223 mvcc.cc:205] Tried to move back new op lower bound from 7332625800182693888 to 7332625800047570944. Current Snapshot: MvccSnapshot[applied={T|T < 7332625800182693888}]
I20260923 19:30:45.784058 10732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:45.784459 10732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:30:45.789090 10732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b00d415ab4c43b0ba44dd0ab4ea4040: Sending DeleteTablet for 3 replicas of tablet dba3678f8b804e3182d4b2673dcd1ce0
I20260923 19:30:45.790558 10856 tablet_service.cc:1514] Processing DeleteTablet for tablet dba3678f8b804e3182d4b2673dcd1ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60954
I20260923 19:30:45.790562 10990 tablet_service.cc:1514] Processing DeleteTablet for tablet dba3678f8b804e3182d4b2673dcd1ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51348
I20260923 19:30:45.790946 11124 tablet_service.cc:1514] Processing DeleteTablet for tablet dba3678f8b804e3182d4b2673dcd1ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:30:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42078
I20260923 19:30:45.792881 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 10793
I20260923 19:30:45.803012 11231 tablet_replica.cc:330] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: stopping tablet replica
I20260923 19:30:45.803022 11233 tablet_replica.cc:330] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: stopping tablet replica
I20260923 19:30:45.803794 11233 raft_consensus.cc:2229] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:45.804533 11233 raft_consensus.cc:2258] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:45.804767 11231 raft_consensus.cc:2229] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:30:45.805263 11231 raft_consensus.cc:2258] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:30:45.807801 11233 ts_tablet_manager.cc:1905] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:45.808007 11231 ts_tablet_manager.cc:1905] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:45.811851 11232 tablet_replica.cc:330] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: stopping tablet replica
I20260923 19:30:45.812352 11232 raft_consensus.cc:2229] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:30:45.813197 11232 raft_consensus.cc:2258] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 19:30:45 dist-test-slave-kxfb krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:30:45.816213 11233 ts_tablet_manager.cc:1918] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:45.816489 11233 log.cc:1196] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/dba3678f8b804e3182d4b2673dcd1ce0
I20260923 19:30:45.817094 11233 ts_tablet_manager.cc:1939] T dba3678f8b804e3182d4b2673dcd1ce0 P 581c67bc0f0c44c6b8d417c9ae9c1899: Deleting consensus metadata
I20260923 19:30:45.820016 11232 ts_tablet_manager.cc:1905] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:30:45.820570 10719 catalog_manager.cc:4841] TS 581c67bc0f0c44c6b8d417c9ae9c1899 (127.0.97.3:43909): tablet dba3678f8b804e3182d4b2673dcd1ce0 (table test-table [id=7fa8a4687d254f9dad1cd151690fd545]) successfully deleted
I20260923 19:30:45.821393 11231 ts_tablet_manager.cc:1918] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:45.821616 11231 log.cc:1196] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/dba3678f8b804e3182d4b2673dcd1ce0
I20260923 19:30:45.822212 11231 ts_tablet_manager.cc:1939] T dba3678f8b804e3182d4b2673dcd1ce0 P e420aef6fcd6454bb1bc283e75dd5612: Deleting consensus metadata
I20260923 19:30:45.824368 10716 catalog_manager.cc:4841] TS e420aef6fcd6454bb1bc283e75dd5612 (127.0.97.1:45255): tablet dba3678f8b804e3182d4b2673dcd1ce0 (table test-table [id=7fa8a4687d254f9dad1cd151690fd545]) successfully deleted
I20260923 19:30:45.825342 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.794505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.0.97.1:45255)
0923 19:30:45.794716 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.794724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.805537 (+ 10813us) server_negotiation.cc:407] Connection header received
0923 19:30:45.805986 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.805992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.806104 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.806181 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:45.807868 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.809049 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.810947 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.811561 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.811658 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.816991 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:45.817010 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:45.817037 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:45.817103 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:45.820904 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:45.821820 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:45.821829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:45.821833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:45.822015 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:45.822530 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:45.822600 (+ 70us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:45.822604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:45.823088 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:45.823423 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.824667 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.824937 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 19:30:45.826117 10916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:45.827539 11232 ts_tablet_manager.cc:1918] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:30:45.827714 11232 log.cc:1196] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/dba3678f8b804e3182d4b2673dcd1ce0
I20260923 19:30:45.828336 11232 ts_tablet_manager.cc:1939] T dba3678f8b804e3182d4b2673dcd1ce0 P a4bf4cebb2624b8a99b8bee1f7807ecf: Deleting consensus metadata
I20260923 19:30:45.830221 10719 catalog_manager.cc:4841] TS a4bf4cebb2624b8a99b8bee1f7807ecf (127.0.97.2:38201): tablet dba3678f8b804e3182d4b2673dcd1ce0 (table test-table [id=7fa8a4687d254f9dad1cd151690fd545]) successfully deleted
I20260923 19:30:45.921702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 10793
I20260923 19:30:45.939150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 10927
Sep 23 19:30:45 dist-test-slave-kxfb krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:30:45.957110 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:45.940017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.0.97.2:38201)
0923 19:30:45.940251 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:45.940258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:45.940351 (+ 93us) server_negotiation.cc:407] Connection header received
0923 19:30:45.940554 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:45.940564 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:45.940642 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:45.940739 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:45.942066 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.943046 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.944309 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:45.944760 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:45.944882 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:45.949606 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:45.949633 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:45.949655 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:45.949723 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:45.953695 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:45.954790 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:45.954800 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:45.954804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:45.954977 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:45.955385 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:45.955390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:45.955394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:45.955847 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:45.956167 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:45.956481 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 19:30:45.956713 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 19:30:45.958017 11050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:46.082116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 10927
I20260923 19:30:46.097812 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 11061
Sep 23 19:30:46 dist-test-slave-kxfb krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:30:46.113306 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:30:46.098783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.0.97.3:43909)
0923 19:30:46.098989 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 19:30:46.098993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:30:46.099015 (+ 22us) server_negotiation.cc:407] Connection header received
0923 19:30:46.099189 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:30:46.099193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:30:46.099263 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:30:46.099346 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:30:46.100768 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:46.101492 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:46.102895 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:30:46.103301 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:30:46.103394 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:30:46.108033 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:30:46.108044 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:30:46.108061 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:30:46.108109 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:30:46.110850 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:46.111550 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:46.111557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:46.111559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:46.111679 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:30:46.111983 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:30:46.111986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:30:46.111988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:30:46.112318 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:30:46.112516 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 19:30:46.112793 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 19:30:46.112969 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 19:30:46.114094 11184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:46.209282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 11061
I20260923 19:30:46.225320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 10700
I20260923 19:30:46.226511 10762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:30:46.333081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 10700
2026-09-23T19:30:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4394 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:30:46 dist-test-slave-kxfb krb5kdc[11241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:30:46 dist-test-slave-kxfb krb5kdc[11241](info): set up 2 sockets
Sep 23 19:30:46 dist-test-slave-kxfb krb5kdc[11241](info): commencing operation
krb5kdc: starting...
W20260923 19:30:47.359447 388 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:30:47 dist-test-slave-kxfb krb5kdc[11241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:30:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260923 19:30:47.722230 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhSGBcM/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/postgres -l logfile start
2026-09-23 19:30:50.246 UTC [11271] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 19:30:50.246 UTC [11271] LOG: listening on IPv4 address "127.0.97.20", port 54795
2026-09-23 19:30:50.250 UTC [11271] LOG: listening on Unix socket "/tmp/.s.PGSQL.54795"
2026-09-23 19:30:50.274 UTC [11274] LOG: database system was shut down at 2026-09-23 19:30:48 UTC
2026-09-23 19:30:50.278 UTC [11271] LOG: database system is ready to accept connections
I20260923 19:30:51.178674 388 mini_postgres.cc:96] Postgres bound to 54795
2026-09-23 19:30:51.198 UTC [11282] FATAL: database "slave" does not exist
127.0.97.20:54795 - accepting connections
I20260923 19:30:51.199661 388 mini_ranger.cc:161] Starting Ranger...
I20260923 19:30:51.231199 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 19:30:51.734668 388 mini_ranger.cc:87] Created ranger Postgres database
I20260923 19:30:51.734848 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-admin
2026-09-23 19:30:52,040 [I] DB FLAVOR :POSTGRES
2026-09-23 19:30:52,041 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 19:30:52,041 [I] Checking connection..
2026-09-23 19:30:52,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 19:30:52,297 [I] Checking connection passed.
2026-09-23 19:30:52,297 [I] --------- Verifying version history table ---------
2026-09-23 19:30:52,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 19:30:52,553 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 19:30:52,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 19:30:52,820 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 19:30:52,820 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 19:30:52,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 19:30:53.079 UTC [11346] WARNING: there is no transaction in progress
2026-09-23 19:30:53,085 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 19:30:53,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 19:30:53,363 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 19:30:53,364 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 19:30:53,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 19:30:53,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 19:30:53,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kxfb.c.gcp-upstream.internal','N') ;"
2026-09-23 19:30:54,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 19:30:54,395 [I] Table x_portal_user does not exist in database ranger
2026-09-23 19:30:54,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 19:30:54,646 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 19:30:54,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 19:30:54,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 19:30:55,147 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 19:30:55,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 19:30:56.143 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.160 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.177 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.193 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.210 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.230 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.285 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.292 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.307 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.317 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.328 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.344 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.355 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.362 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.369 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:56.376 UTC [11454] WARNING: there is no transaction in progress
2026-09-23 19:30:57,038 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 19:30:57,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 19:30:57,277 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 19:30:57,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 19:30:57,549 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 19:30:57,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 19:30:57,815 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 19:30:57,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 19:30:58,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 19:30:58,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kxfb.c.gcp-upstream.internal';"
2026-09-23 19:30:58,598 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 19:30:58,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 19:30:58,859 [I] DB_PATCHES have already been applied
I20260923 19:30:58.865438 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhSGBcM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 19:30:58.865579 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260923 19:30:58.883553 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:37253
Sep 23, 2026 7:30:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 7:30:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 7:30:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 7:30:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 7:30:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 7:30:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 7:30:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 19:30:59.929925 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.730s user 0.003s sys 0.053s
Sep 23, 2026 7:31:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 7:31:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 7:31:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 7:31:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 7:31:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 19:31:21.446400 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 19:31:21 dist-test-slave-kxfb krb5kdc[11241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191881, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 19:31:21.632843 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 19:31:21.652294 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 19:31:22.524170 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 19:31:22.524318 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-kms
I20260923 19:31:22.524483 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:54795
2026-09-23 19:31:22,956 [I] DB FLAVOR :POSTGRES
2026-09-23 19:31:22,956 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 19:31:22,956 [I] Checking connection
2026-09-23 19:31:22,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 19:31:23,202 [I] connection success
2026-09-23 19:31:23,203 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 19:31:23,203 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 19:31:23,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 19:31:23,447 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 19:31:23,448 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 19:31:23,448 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 19:31:23,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 19:31:23,740 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 19:31:24.229986 388 mini_ranger_kms.cc:310] Created kms service
I20260923 19:31:24.417675 388 mini_ranger_kms.cc:326] Created kudu user
I20260923 19:31:24.458326 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 19:31:24 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191881, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:31:24.868474 388 mini_ranger_kms.cc:384] Added ranger policy
I20260923 19:31:24.868801 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 19:31:24.868861 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260923 19:31:24.891852 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11634
I20260923 19:31:24.892095 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:58233
Sep 23, 2026 7:31:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 7:31:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 7:31:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 7:31:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 7:31:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 7:31:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 19:31:27 dist-test-slave-kxfb krb5kdc[11241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790191887, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:31:27.061107 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.428s user 0.122s sys 0.344s
I20260923 19:31:27.061367 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 19:31:27.061586 388 mini_ranger_kms.cc:401] 127.0.97.20:58233/kms/v1/keys
Sep 23 19:31:27 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191887, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 19:31:27 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191887, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 19:31:30 dist-test-slave-kxfb krb5kdc[11241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:31:30.378912 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42221
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42221
--ranger_config_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58233/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:31:30.778671 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:31:30.779083 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:31:30.779225 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:31:30.787577 11685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:31:30.787729 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:31:30.787808 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:31:30.787882 11685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:31:30.787947 11685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:31:30.799512 11685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:58233/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42221
--ranger_config_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:31:30.801139 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:31:30.802753 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:31:30.815829 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:31:30.816450 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:31:30.816423 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:31:30.817030 11685 server_base.cc:1034] running on GCE node
I20260923 19:31:30.817910 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:31:30.819576 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:31:30.820806 11685 hybrid_clock.cc:648] HybridClock initialized: now 1790191890820735 us; error 60 us; skew 500 ppm
I20260923 19:31:30.821192 11685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:31:30 dist-test-slave-kxfb krb5kdc[11241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:31:30.835979 11685 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:31:30.837743 11685 webserver.cc:460] Webserver started at http://127.0.97.62:34653/ using document root <none> and password file <none>
I20260923 19:31:30.838551 11685 fs_manager.cc:359] Metadata directory not provided
I20260923 19:31:30.838666 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:31:30.838961 11685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 19:31:30 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:31:30.893172 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "324c65d951ae415c8e2ba57fffd49b38"
format_stamp: "Formatted at 2026-09-23 19:31:30 on dist-test-slave-kxfb"
server_key: "e963564df5d7d292e84588ab47d983d4"
server_key_iv: "89b664a2a493b05e401feedde74d9dec"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:30.894098 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "324c65d951ae415c8e2ba57fffd49b38"
format_stamp: "Formatted at 2026-09-23 19:31:30 on dist-test-slave-kxfb"
server_key: "e963564df5d7d292e84588ab47d983d4"
server_key_iv: "89b664a2a493b05e401feedde74d9dec"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:30.900609 11685 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 19:31:30.918385 11700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:31:30.920600 11685 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260923 19:31:30.920843 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "324c65d951ae415c8e2ba57fffd49b38"
format_stamp: "Formatted at 2026-09-23 19:31:30 on dist-test-slave-kxfb"
server_key: "e963564df5d7d292e84588ab47d983d4"
server_key_iv: "89b664a2a493b05e401feedde74d9dec"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:30.921139 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:31:30.943711 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:31:30.948932 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:31:30.949277 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:31:30.972330 11685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42221
I20260923 19:31:30.972358 11751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42221 every 8 connection(s)
I20260923 19:31:30.974764 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:31:30.979665 11752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:31:30.983327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 11685
I20260923 19:31:30.983536 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:31:30.984289 388 external_mini_cluster.cc:1428] Setting key c3497c67dffdf8b8c26fa2816df3a9fe
I20260923 19:31:30.994912 11752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38: Bootstrap starting.
I20260923 19:31:31.000227 11752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38: Neither blocks nor log segments found. Creating new log.
I20260923 19:31:31.002883 11752 log.cc:824] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38: Log is configured to *not* fsync() on all Append() calls
I20260923 19:31:31.007133 11752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38: No bootstrap required, opened a new log
Sep 23 19:31:31 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:31:31.017896 11752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324c65d951ae415c8e2ba57fffd49b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42221 } }
I20260923 19:31:31.018522 11752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:31:31.018687 11752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324c65d951ae415c8e2ba57fffd49b38, State: Initialized, Role: FOLLOWER
I20260923 19:31:31.019486 11752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324c65d951ae415c8e2ba57fffd49b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42221 } }
I20260923 19:31:31.019783 11752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:31:31.019927 11752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:31:31.020135 11752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:31:31.022912 11752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324c65d951ae415c8e2ba57fffd49b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42221 } }
I20260923 19:31:31.023521 11752 leader_election.cc:302] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 324c65d951ae415c8e2ba57fffd49b38; no voters:
I20260923 19:31:31.024264 11752 leader_election.cc:288] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:31:31.024479 11757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:31:31.030329 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:30.986815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.0.97.62:42221)
0923 19:31:31.002151 (+ 15336us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:31.002170 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:31.002230 (+ 60us) server_negotiation.cc:407] Connection header received
0923 19:31:31.004733 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:31.004765 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:31.005362 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:31.005875 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:31:31.007552 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:31.009636 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:31.011624 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:31.012772 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:31.012988 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:31.018689 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:31:31.018704 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:31:31.018732 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:31:31.018812 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:31:31.023342 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:31.024487 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:31.024499 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:31.024509 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:31.024757 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:31.025313 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:31.025333 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:31.025337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:31.026400 (+ 1063us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:31:31.026769 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:31.027858 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0923 19:31:31.028450 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15054,"thread_start_us":14801,"threads_started":1}
I20260923 19:31:31.032174 11757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [term 1 LEADER]: Becoming Leader. State: Replica: 324c65d951ae415c8e2ba57fffd49b38, State: Running, Role: LEADER
I20260923 19:31:31.033074 11757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324c65d951ae415c8e2ba57fffd49b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42221 } }
I20260923 19:31:31.036536 11752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:31:31.039098 11758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324c65d951ae415c8e2ba57fffd49b38. Latest consensus state: current_term: 1 leader_uuid: "324c65d951ae415c8e2ba57fffd49b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324c65d951ae415c8e2ba57fffd49b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42221 } } }
I20260923 19:31:31.040352 11758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [sys.catalog]: This master's current role is: LEADER
I20260923 19:31:31.038424 11759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "324c65d951ae415c8e2ba57fffd49b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324c65d951ae415c8e2ba57fffd49b38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42221 } } }
I20260923 19:31:31.041770 11759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38 [sys.catalog]: This master's current role is: LEADER
I20260923 19:31:31.049445 11762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:31:31.056124 11762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:31:31.064642 11752 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 19:31:31.076242 11762 catalog_manager.cc:1347] Generated new cluster ID: 1eb6de48d2584b21b34b9102ac9e0fb7
I20260923 19:31:31.077215 11762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:31:31.128319 11762 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:31:31.130612 11762 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:31:31.143105 11762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38: Generated new TSK 0
I20260923 19:31:31.144709 11762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:31:31.878911 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42221
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:31:32 dist-test-slave-kxfb krb5kdc[11241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790191892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 19:31:32.372056 11797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:31:32.372470 11797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:31:32.372622 11797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:31:32.384773 11797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:31:32.384953 11797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:31:32.385217 11797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:31:32.401839 11797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:58233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:31:32.403724 11797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:31:32.405501 11797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:31:32.426834 11808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:31:32.427656 11807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:31:32.431146 11797 server_base.cc:1034] running on GCE node
W20260923 19:31:32.431385 11810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:31:32.432224 11797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:31:32.433676 11797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:31:32.434955 11797 hybrid_clock.cc:648] HybridClock initialized: now 1790191892434892 us; error 51 us; skew 500 ppm
I20260923 19:31:32.435380 11797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:31:32 dist-test-slave-kxfb krb5kdc[11241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:31:32.438966 11797 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:31:32.444425 11797 webserver.cc:460] Webserver started at http://127.0.97.1:34251/ using document root <none> and password file <none>
I20260923 19:31:32.445685 11797 fs_manager.cc:359] Metadata directory not provided
I20260923 19:31:32.445833 11797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:31:32.446189 11797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 19:31:32 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:31:32.466203 11797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "4c1cf27a24d34a35b51b3633d90fcf10"
format_stamp: "Formatted at 2026-09-23 19:31:32 on dist-test-slave-kxfb"
server_key: "599041f97affdd14f572b8d4e7cf1dc3"
server_key_iv: "08807a5ed2fd41b5484a612bd419ae59"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:32.467383 11797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4c1cf27a24d34a35b51b3633d90fcf10"
format_stamp: "Formatted at 2026-09-23 19:31:32 on dist-test-slave-kxfb"
server_key: "599041f97affdd14f572b8d4e7cf1dc3"
server_key_iv: "08807a5ed2fd41b5484a612bd419ae59"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:32.476233 11797 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260923 19:31:32.506708 11817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:31:32.509829 11797 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.001s
I20260923 19:31:32.510111 11797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "4c1cf27a24d34a35b51b3633d90fcf10"
format_stamp: "Formatted at 2026-09-23 19:31:32 on dist-test-slave-kxfb"
server_key: "599041f97affdd14f572b8d4e7cf1dc3"
server_key_iv: "08807a5ed2fd41b5484a612bd419ae59"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:32.510474 11797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:31:32.560721 11797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:31:32.567289 11797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:31:32.567620 11797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:31:32.569024 11797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:31:32.572516 11797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:31:32.572608 11797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:31:32.572774 11797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:31:32.572831 11797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 19:31:32 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:31:32.713989 11797 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38261
I20260923 19:31:32.714059 11929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38261 every 8 connection(s)
I20260923 19:31:32.715760 11797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:31:32.716300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 11797
I20260923 19:31:32.716521 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:31:32.717348 388 external_mini_cluster.cc:1428] Setting key 73ba6bd350d5f73edf5892fecde537e9
Sep 23 19:31:32 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:31:32.793429 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:32.721833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42901 (local address 127.0.97.62:42221)
0923 19:31:32.758882 (+ 37049us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:32.758906 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:32.761614 (+ 2708us) server_negotiation.cc:407] Connection header received
0923 19:31:32.773300 (+ 11686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:32.773308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:32.773435 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:32.773524 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:31:32.777027 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:32.778224 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:32.780653 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:32.781249 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:32.781341 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:32.786428 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:31:32.786436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:31:32.786441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:31:32.786481 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:31:32.789767 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:32.790696 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:32.790702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:32.790705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:32.790848 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:32.791366 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:32.791371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:32.791375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:32.791709 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:31:32.791837 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:32.792794 (+ 957us) server_negotiation.cc:299] Negotiation successful
0923 19:31:32.793029 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36886,"thread_start_us":36163,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 19:31:32.795892 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:32.720679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42221 (local address 127.0.97.1:42901)
0923 19:31:32.760939 (+ 40260us) negotiation.cc:107] Waiting for socket to connect
0923 19:31:32.761018 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0923 19:31:32.772824 (+ 11806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:31:32.774200 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:31:32.774214 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:31:32.775002 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:31:32.775975 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:32.776001 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:32.778461 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:32.778466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:32.780493 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:32.780509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:32.781695 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:32.781702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:32.781997 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:32.782906 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:31:32.782949 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:31:32.786144 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:31:32.789927 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:31:32.789937 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:31:32.789956 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:31:32.790557 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:31:32.790992 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:31:32.790997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:31:32.791000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:31:32.791241 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:31:32.791939 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:31:32.791947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:31:32.792363 (+ 416us) client_negotiation.cc:769] Sending connection context
0923 19:31:32.794009 (+ 1646us) client_negotiation.cc:240] Negotiation successful
0923 19:31:32.794551 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39854,"thread_start_us":39522,"threads_started":1}
I20260923 19:31:32.797986 11930 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42221
I20260923 19:31:32.798317 11930 heartbeater.cc:464] Registering TS with master...
I20260923 19:31:32.799131 11930 heartbeater.cc:511] Master 127.0.97.62:42221 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 19:31:32.801510 11717 ts_manager.cc:194] Registered new tserver with Master: 4c1cf27a24d34a35b51b3633d90fcf10 (127.0.97.1:38261)
I20260923 19:31:32.803403 11717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260923 19:31:32.840725 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.117s user 0.000s sys 0.019s
I20260923 19:31:32.841576 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42221
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:31:33 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260923 19:31:33.328941 11936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:31:33.329442 11936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:31:33.329639 11936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:31:33.341183 11936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:31:33.341439 11936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:31:33.341672 11936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:31:33.357723 11936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:58233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:31:33.359726 11936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:31:33.361476 11936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:31:33.397830 11943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:31:33.405966 11944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:31:33.420646 11947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:31:33.425056 11936 server_base.cc:1034] running on GCE node
I20260923 19:31:33.425810 11936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:31:33.436816 11936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:31:33.438733 11936 hybrid_clock.cc:648] HybridClock initialized: now 1790191893438615 us; error 132 us; skew 500 ppm
I20260923 19:31:33.439280 11936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:31:33 dist-test-slave-kxfb krb5kdc[11241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:31:33.443056 11936 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:31:33.445279 11936 webserver.cc:460] Webserver started at http://127.0.97.2:35685/ using document root <none> and password file <none>
I20260923 19:31:33.446069 11936 fs_manager.cc:359] Metadata directory not provided
I20260923 19:31:33.446188 11936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:31:33.446547 11936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 19:31:33 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:31:33.470201 11936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "61c7041daf434135a0a7ca18885150d6"
format_stamp: "Formatted at 2026-09-23 19:31:33 on dist-test-slave-kxfb"
server_key: "37d80483e024a4ab25d92b343a62641a"
server_key_iv: "b17717d26d374be063c490a9c4bdad7b"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:33.471498 11936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "61c7041daf434135a0a7ca18885150d6"
format_stamp: "Formatted at 2026-09-23 19:31:33 on dist-test-slave-kxfb"
server_key: "37d80483e024a4ab25d92b343a62641a"
server_key_iv: "b17717d26d374be063c490a9c4bdad7b"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:33.477921 11936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:31:33.504757 11960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:31:33.506249 11936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 19:31:33.506480 11936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "61c7041daf434135a0a7ca18885150d6"
format_stamp: "Formatted at 2026-09-23 19:31:33 on dist-test-slave-kxfb"
server_key: "37d80483e024a4ab25d92b343a62641a"
server_key_iv: "b17717d26d374be063c490a9c4bdad7b"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:33.506688 11936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:31:33.523798 11936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:31:33.528406 11936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:31:33.528702 11936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:31:33.529703 11936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:31:33.531414 11936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:31:33.531493 11936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:31:33.531616 11936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:31:33.531668 11936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:31:33.578047 12072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41425 every 8 connection(s)
I20260923 19:31:33.578460 11936 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41425
I20260923 19:31:33.579650 11936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:31:33.583145 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 11936
I20260923 19:31:33.583310 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:31:33.583804 388 external_mini_cluster.cc:1428] Setting key 1df22ea9ca0e8e810ff3011e10484e30
Sep 23 19:31:33 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 19:31:33.620711 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:33.586014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35029 (local address 127.0.97.62:42221)
0923 19:31:33.594740 (+ 8726us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:33.594749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:33.596314 (+ 1565us) server_negotiation.cc:407] Connection header received
0923 19:31:33.599051 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:33.599058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:33.599196 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:33.599288 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:31:33.601925 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:33.603219 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:33.605856 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:33.606534 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:33.606634 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:33.611735 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:31:33.611742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:31:33.611747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:31:33.611791 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:31:33.616529 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:33.617804 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:33.617809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:33.617814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:33.617970 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:33.618543 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:33.618548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:33.618552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:33.618933 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:31:33.619045 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:33.620080 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0923 19:31:33.620328 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8549,"thread_start_us":9171,"threads_started":1}
I20260923 19:31:33.623739 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:33.585500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42221 (local address 127.0.97.2:35029)
0923 19:31:33.595219 (+ 9719us) negotiation.cc:107] Waiting for socket to connect
0923 19:31:33.595296 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0923 19:31:33.598621 (+ 3325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:31:33.599921 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:31:33.599935 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:31:33.600645 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:31:33.601595 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:33.601719 (+ 124us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:33.603448 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:33.603455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:33.605680 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:33.605704 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:33.607007 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:33.607013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:33.607339 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:33.608231 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:31:33.608264 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:31:33.611451 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:31:33.616802 (+ 5351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:31:33.616815 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:31:33.616841 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:31:33.617619 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:31:33.618110 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:31:33.618116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:31:33.618120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:31:33.618418 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:31:33.619154 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:31:33.619163 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:31:33.619654 (+ 491us) client_negotiation.cc:769] Sending connection context
0923 19:31:33.621401 (+ 1747us) client_negotiation.cc:240] Negotiation successful
0923 19:31:33.622084 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9307,"thread_start_us":8512,"threads_started":1}
I20260923 19:31:33.626256 12073 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42221
I20260923 19:31:33.626747 12073 heartbeater.cc:464] Registering TS with master...
I20260923 19:31:33.627712 12073 heartbeater.cc:511] Master 127.0.97.62:42221 requested a full tablet report, sending...
I20260923 19:31:33.629949 11717 ts_manager.cc:194] Registered new tserver with Master: 61c7041daf434135a0a7ca18885150d6 (127.0.97.2:41425)
I20260923 19:31:33.631274 11717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:31:33.653501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42221
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:31:33.806942 11930 heartbeater.cc:503] Master 127.0.97.62:42221 was elected leader, sending a full tablet report...
W20260923 19:31:34.017194 12079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:31:34.017537 12079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:31:34.017648 12079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:31:34.024994 12079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:31:34.025131 12079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:31:34.025280 12079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:31:34.035408 12079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:58233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:31:34.036720 12079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:31:34.038000 12079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:31:34.052215 12085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:31:34.052099 12084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:31:34.052223 12087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:31:34.055806 12079 server_base.cc:1034] running on GCE node
I20260923 19:31:34.056504 12079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:31:34.057504 12079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:31:34.058710 12079 hybrid_clock.cc:648] HybridClock initialized: now 1790191894058635 us; error 72 us; skew 500 ppm
I20260923 19:31:34.059037 12079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:31:34 dist-test-slave-kxfb krb5kdc[11241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:31:34.062223 12079 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:31:34.063776 12079 webserver.cc:460] Webserver started at http://127.0.97.3:46097/ using document root <none> and password file <none>
I20260923 19:31:34.064436 12079 fs_manager.cc:359] Metadata directory not provided
I20260923 19:31:34.064539 12079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:31:34.064801 12079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 19:31:34 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:31:34.080490 12079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "4495af7d667544baa893e5026728c14d"
format_stamp: "Formatted at 2026-09-23 19:31:34 on dist-test-slave-kxfb"
server_key: "172146463d964993d254eb1d8f883c9c"
server_key_iv: "710c394a0a6a744069a37cf85a413bd0"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:34.081619 12079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4495af7d667544baa893e5026728c14d"
format_stamp: "Formatted at 2026-09-23 19:31:34 on dist-test-slave-kxfb"
server_key: "172146463d964993d254eb1d8f883c9c"
server_key_iv: "710c394a0a6a744069a37cf85a413bd0"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:34.088784 12079 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260923 19:31:34.108852 12094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:31:34.110867 12079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:31:34.111086 12079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "4495af7d667544baa893e5026728c14d"
format_stamp: "Formatted at 2026-09-23 19:31:34 on dist-test-slave-kxfb"
server_key: "172146463d964993d254eb1d8f883c9c"
server_key_iv: "710c394a0a6a744069a37cf85a413bd0"
server_key_version: "kuduclusterkey@0"
I20260923 19:31:34.111351 12079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:31:34.128158 12079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:31:34.131920 12079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:31:34.132161 12079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:31:34.133147 12079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:31:34.134567 12079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:31:34.134640 12079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:31:34.134732 12079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:31:34.134784 12079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:31:34.172605 12079 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33735
I20260923 19:31:34.172770 12206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33735 every 8 connection(s)
I20260923 19:31:34.173862 12079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:31:34.181632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 12079
I20260923 19:31:34.181810 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:31:34.182353 388 external_mini_cluster.cc:1428] Setting key 3d0b6c6c17bc63b9f87ec137a5a216b6
Sep 23 19:31:34 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:31:34.195449 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.176888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39163 (local address 127.0.97.62:42221)
0923 19:31:34.177522 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.177530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.178647 (+ 1117us) server_negotiation.cc:407] Connection header received
0923 19:31:34.180456 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.180461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.180549 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.180687 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:31:34.182144 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.183231 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.185010 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.185681 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.185795 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.189378 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:31:34.189384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:31:34.189388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:31:34.189427 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:31:34.192220 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.192992 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.192997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.192999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.193156 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.193630 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.193637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.193639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.194003 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:31:34.194172 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.194882 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.195116 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":247,"threads_started":1}
I20260923 19:31:34.196218 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.177532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42221 (local address 127.0.97.3:39163)
0923 19:31:34.178451 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0923 19:31:34.178503 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 19:31:34.180150 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:31:34.180895 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:31:34.180903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:31:34.181314 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:31:34.181959 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.181978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.183388 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.183393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.184833 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.184850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.185888 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.185893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.186084 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.186821 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:31:34.186843 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:31:34.189141 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:31:34.192418 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:31:34.192425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:31:34.192439 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:31:34.192877 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:31:34.193253 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:31:34.193256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:31:34.193262 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:31:34.193485 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:31:34.194126 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:31:34.194131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:31:34.194508 (+ 377us) client_negotiation.cc:769] Sending connection context
0923 19:31:34.194744 (+ 236us) client_negotiation.cc:240] Negotiation successful
0923 19:31:34.195118 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":355,"threads_started":1}
I20260923 19:31:34.198205 12207 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42221
I20260923 19:31:34.198661 12207 heartbeater.cc:464] Registering TS with master...
I20260923 19:31:34.199443 12207 heartbeater.cc:511] Master 127.0.97.62:42221 requested a full tablet report, sending...
I20260923 19:31:34.201193 11716 ts_manager.cc:194] Registered new tserver with Master: 4495af7d667544baa893e5026728c14d (127.0.97.3:33735)
I20260923 19:31:34.202323 11716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39163
I20260923 19:31:34.212143 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:31:34.232172 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.217376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.97.62:42221)
0923 19:31:34.217764 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.217771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.218133 (+ 362us) server_negotiation.cc:407] Connection header received
0923 19:31:34.218392 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.218397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.218508 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.218609 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:31:34.220074 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.220846 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.223292 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.223763 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.223882 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.226647 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:31:34.226653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:31:34.226658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:31:34.226700 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:31:34.229246 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.230053 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.230057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.230060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.230202 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.230645 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.230649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.230656 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.230977 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:31:34.231190 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.231639 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.231830 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260923 19:31:34.237123 11716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:31:34.245117 11716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:31:34.291935 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.261418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.0.97.2:41425)
0923 19:31:34.281628 (+ 20210us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.281639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.282514 (+ 875us) server_negotiation.cc:407] Connection header received
0923 19:31:34.282636 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.282645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.282935 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.283150 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:31:34.285283 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.286479 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.289613 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.290774 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.290905 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.291123 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.291440 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.291610 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20024,"thread_start_us":19881,"threads_started":1}
I20260923 19:31:34.291996 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.260941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41425 (local address 127.0.0.1:42286)
0923 19:31:34.282123 (+ 21182us) negotiation.cc:107] Waiting for socket to connect
0923 19:31:34.282146 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:31:34.282374 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:31:34.283399 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:31:34.283403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:31:34.283425 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:31:34.285020 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.285035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.286620 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.286626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.289407 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.289416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.290990 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.290997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.291221 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.291258 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 19:31:34.291373 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 19:31:34.291592 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21033,"mutex_wait_us":72,"thread_start_us":20887,"threads_started":1}
I20260923 19:31:34.293156 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.260424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33735 (local address 127.0.0.1:36582)
0923 19:31:34.281061 (+ 20637us) negotiation.cc:107] Waiting for socket to connect
0923 19:31:34.281086 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:31:34.281266 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:31:34.283218 (+ 1952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:31:34.283224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:31:34.283269 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:31:34.285081 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.285091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.287405 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.287411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.289639 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.289654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.292219 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.292226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.292402 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.292439 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 19:31:34.292538 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 19:31:34.292707 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20464,"mutex_wait_us":147,"spinlock_wait_cycles":7680,"thread_start_us":9673,"threads_started":1}
I20260923 19:31:34.293190 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.261238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.0.97.3:33735)
0923 19:31:34.280215 (+ 18977us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.280229 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.281429 (+ 1200us) server_negotiation.cc:407] Connection header received
0923 19:31:34.281576 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.281589 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.282635 (+ 1046us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.282893 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:31:34.285312 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.286983 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.289827 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.291479 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.291682 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.292056 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.292632 (+ 576us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.292843 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18686,"thread_start_us":18989,"threads_started":1}
I20260923 19:31:34.293756 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.260092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38261 (local address 127.0.0.1:49584)
0923 19:31:34.270640 (+ 10548us) negotiation.cc:107] Waiting for socket to connect
0923 19:31:34.270689 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 19:31:34.270942 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:31:34.283203 (+ 12261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:31:34.283211 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:31:34.283264 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:31:34.284978 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.284995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.287129 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.287135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.289397 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.289415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.292775 (+ 3360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.292781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.292968 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.293001 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 19:31:34.293085 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 19:31:34.293234 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10220,"mutex_wait_us":66,"spinlock_wait_cycles":14720,"thread_start_us":20752,"threads_started":1}
I20260923 19:31:34.296129 12008 tablet_service.cc:1467] Processing CreateTablet for tablet ce90a4411126470090149f9a40f77290 (DEFAULT_TABLE table=test-table [id=00ef578cd0f949e6a8c4cbf2a7444459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:31:34.297326 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce90a4411126470090149f9a40f77290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:31:34.297232 12142 tablet_service.cc:1467] Processing CreateTablet for tablet ce90a4411126470090149f9a40f77290 (DEFAULT_TABLE table=test-table [id=00ef578cd0f949e6a8c4cbf2a7444459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:31:34.298633 12142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce90a4411126470090149f9a40f77290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:31:34.301688 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.260372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.0.97.1:38261)
0923 19:31:34.282547 (+ 22175us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.282556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.282581 (+ 25us) server_negotiation.cc:407] Connection header received
0923 19:31:34.282673 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.282679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.282934 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.283637 (+ 703us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:31:34.285185 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.286880 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.289638 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.292266 (+ 2628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.292401 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.292610 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.294272 (+ 1662us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.294493 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21985,"thread_start_us":21802,"threads_started":1}
I20260923 19:31:34.306259 11865 tablet_service.cc:1467] Processing CreateTablet for tablet ce90a4411126470090149f9a40f77290 (DEFAULT_TABLE table=test-table [id=00ef578cd0f949e6a8c4cbf2a7444459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:31:34.308019 11865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce90a4411126470090149f9a40f77290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:31:34.309051 12226 tablet_bootstrap.cc:492] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: Bootstrap starting.
I20260923 19:31:34.312623 12226 tablet_bootstrap.cc:654] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: Neither blocks nor log segments found. Creating new log.
I20260923 19:31:34.312909 12227 tablet_bootstrap.cc:492] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: Bootstrap starting.
I20260923 19:31:34.314558 12226 log.cc:824] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: Log is configured to *not* fsync() on all Append() calls
I20260923 19:31:34.316756 12227 tablet_bootstrap.cc:654] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: Neither blocks nor log segments found. Creating new log.
I20260923 19:31:34.318007 12227 log.cc:824] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: Log is configured to *not* fsync() on all Append() calls
I20260923 19:31:34.318321 12226 tablet_bootstrap.cc:492] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: No bootstrap required, opened a new log
I20260923 19:31:34.318645 12226 ts_tablet_manager.cc:1397] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260923 19:31:34.321116 12227 tablet_bootstrap.cc:492] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: No bootstrap required, opened a new log
I20260923 19:31:34.321372 12227 ts_tablet_manager.cc:1397] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 19:31:34.325475 12230 tablet_bootstrap.cc:492] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: Bootstrap starting.
I20260923 19:31:34.328794 12226 raft_consensus.cc:348] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260923 19:31:34.329265 12226 raft_consensus.cc:374] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:31:34.329423 12226 raft_consensus.cc:729] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c7041daf434135a0a7ca18885150d6, State: Initialized, Role: FOLLOWER
I20260923 19:31:34.330257 12230 tablet_bootstrap.cc:654] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: Neither blocks nor log segments found. Creating new log.
I20260923 19:31:34.330113 12226 consensus_queue.cc:260] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260923 19:31:34.331502 12073 heartbeater.cc:503] Master 127.0.97.62:42221 was elected leader, sending a full tablet report...
I20260923 19:31:34.331655 12230 log.cc:824] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: Log is configured to *not* fsync() on all Append() calls
I20260923 19:31:34.332156 12226 ts_tablet_manager.cc:1428] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: Time spent starting tablet: real 0.013s user 0.003s sys 0.009s
I20260923 19:31:34.332962 12227 raft_consensus.cc:348] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260923 19:31:34.333539 12227 raft_consensus.cc:374] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:31:34.333688 12227 raft_consensus.cc:729] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4495af7d667544baa893e5026728c14d, State: Initialized, Role: FOLLOWER
I20260923 19:31:34.334435 12227 consensus_queue.cc:260] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260923 19:31:34.335038 12230 tablet_bootstrap.cc:492] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: No bootstrap required, opened a new log
I20260923 19:31:34.335412 12230 ts_tablet_manager.cc:1397] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260923 19:31:34.335669 12207 heartbeater.cc:503] Master 127.0.97.62:42221 was elected leader, sending a full tablet report...
I20260923 19:31:34.337388 12227 ts_tablet_manager.cc:1428] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260923 19:31:34.339432 12231 raft_consensus.cc:482] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:31:34.339737 12231 raft_consensus.cc:504] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260923 19:31:34.341436 12231 leader_election.cc:288] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1cf27a24d34a35b51b3633d90fcf10 (127.0.97.1:38261), 4495af7d667544baa893e5026728c14d (127.0.97.3:33735)
I20260923 19:31:34.345654 12230 raft_consensus.cc:348] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260923 19:31:34.346163 12230 raft_consensus.cc:374] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:31:34.346328 12230 raft_consensus.cc:729] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1cf27a24d34a35b51b3633d90fcf10, State: Initialized, Role: FOLLOWER
W20260923 19:31:34.347294 12074 tablet.cc:2369] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:31:34.347211 12230 consensus_queue.cc:260] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260923 19:31:34.349277 12230 ts_tablet_manager.cc:1428] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260923 19:31:34.351922 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.342863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50265 (local address 127.0.97.3:33735)
0923 19:31:34.343100 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.343106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.344291 (+ 1185us) server_negotiation.cc:407] Connection header received
0923 19:31:34.344652 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.344658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.344737 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.344815 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:31:34.345370 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.346417 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.349724 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.351019 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.351140 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.351248 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.351571 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.351712 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 19:31:34.352778 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.342151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33735 (local address 127.0.97.2:50265)
0923 19:31:34.344164 (+ 2013us) negotiation.cc:107] Waiting for socket to connect
0923 19:31:34.344193 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 19:31:34.344424 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:31:34.344846 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:31:34.344851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:31:34.344876 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:31:34.345189 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.345197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.346607 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.346613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.349490 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.349505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.351238 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.351245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.351401 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.351425 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 19:31:34.352169 (+ 744us) client_negotiation.cc:240] Negotiation successful
0923 19:31:34.352331 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1821,"mutex_wait_us":99,"thread_start_us":1904,"threads_started":1}
I20260923 19:31:34.353343 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.342162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39965 (local address 127.0.97.1:38261)
0923 19:31:34.342568 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.342577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.344530 (+ 1953us) server_negotiation.cc:407] Connection header received
0923 19:31:34.344756 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.344762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.344874 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.344972 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:31:34.345872 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.346981 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.350948 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.352317 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.352438 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.352553 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.352943 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.353096 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":203}
I20260923 19:31:34.354099 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.343194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38261 (local address 127.0.97.2:39965)
0923 19:31:34.344417 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0923 19:31:34.344435 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:31:34.344577 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:31:34.345166 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:31:34.345176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:31:34.345209 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:31:34.345677 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.345691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.348420 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.348426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.350736 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:31:34.350753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.352553 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.352560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:31:34.352755 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.352786 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 19:31:34.353536 (+ 750us) client_negotiation.cc:240] Negotiation successful
0923 19:31:34.353718 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":33,"thread_start_us":447,"threads_started":1}
I20260923 19:31:34.354041 12162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce90a4411126470090149f9a40f77290" candidate_uuid: "61c7041daf434135a0a7ca18885150d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4495af7d667544baa893e5026728c14d" is_pre_election: true
I20260923 19:31:34.354725 12162 raft_consensus.cc:2454] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61c7041daf434135a0a7ca18885150d6 in term 0.
I20260923 19:31:34.355166 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce90a4411126470090149f9a40f77290" candidate_uuid: "61c7041daf434135a0a7ca18885150d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" is_pre_election: true
I20260923 19:31:34.355648 11885 raft_consensus.cc:2454] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61c7041daf434135a0a7ca18885150d6 in term 0.
I20260923 19:31:34.355661 11961 leader_election.cc:302] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4495af7d667544baa893e5026728c14d, 61c7041daf434135a0a7ca18885150d6; no voters:
I20260923 19:31:34.356083 12231 raft_consensus.cc:2781] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:31:34.356277 12231 raft_consensus.cc:482] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:31:34.356407 12231 raft_consensus.cc:3037] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:31:34.358906 12231 raft_consensus.cc:504] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260923 19:31:34.359573 12231 leader_election.cc:288] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1cf27a24d34a35b51b3633d90fcf10 (127.0.97.1:38261), 4495af7d667544baa893e5026728c14d (127.0.97.3:33735)
I20260923 19:31:34.360078 11885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce90a4411126470090149f9a40f77290" candidate_uuid: "61c7041daf434135a0a7ca18885150d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1cf27a24d34a35b51b3633d90fcf10"
I20260923 19:31:34.360268 11885 raft_consensus.cc:3037] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:31:34.360249 12162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce90a4411126470090149f9a40f77290" candidate_uuid: "61c7041daf434135a0a7ca18885150d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4495af7d667544baa893e5026728c14d"
I20260923 19:31:34.360529 12162 raft_consensus.cc:3037] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:31:34.363286 11885 raft_consensus.cc:2454] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61c7041daf434135a0a7ca18885150d6 in term 1.
I20260923 19:31:34.363392 12162 raft_consensus.cc:2454] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61c7041daf434135a0a7ca18885150d6 in term 1.
I20260923 19:31:34.363736 11963 leader_election.cc:302] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1cf27a24d34a35b51b3633d90fcf10, 61c7041daf434135a0a7ca18885150d6; no voters:
I20260923 19:31:34.364073 12231 raft_consensus.cc:2781] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:31:34.364650 12231 raft_consensus.cc:686] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 1 LEADER]: Becoming Leader. State: Replica: 61c7041daf434135a0a7ca18885150d6, State: Running, Role: LEADER
I20260923 19:31:34.365272 12231 consensus_queue.cc:237] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260923 19:31:34.370499 11716 catalog_manager.cc:5495] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c7041daf434135a0a7ca18885150d6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "61c7041daf434135a0a7ca18885150d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61c7041daf434135a0a7ca18885150d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } health_report { overall_health: HEALTHY } } }
I20260923 19:31:34.390255 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.384700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.0.97.2:41425)
0923 19:31:34.384941 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.384947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.384966 (+ 19us) server_negotiation.cc:407] Connection header received
0923 19:31:34.385056 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.385060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.385156 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.385291 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:31:34.385715 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.386778 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.388252 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.388634 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.388727 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.389165 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:31:34.389698 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:31:34.389817 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.389907 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.389976 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 19:31:34.399786 12162 raft_consensus.cc:1261] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [term 1 FOLLOWER]: Refusing update from remote peer 61c7041daf434135a0a7ca18885150d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:31:34.399789 11885 raft_consensus.cc:1261] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [term 1 FOLLOWER]: Refusing update from remote peer 61c7041daf434135a0a7ca18885150d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:31:34.401059 12237 consensus_queue.cc:1035] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1cf27a24d34a35b51b3633d90fcf10" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:31:34.401444 12231 consensus_queue.cc:1035] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4495af7d667544baa893e5026728c14d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:31:34.431005 11715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:31:34.431298 11715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:31:34.435250 11715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 324c65d951ae415c8e2ba57fffd49b38: Sending DeleteTablet for 3 replicas of tablet ce90a4411126470090149f9a40f77290
I20260923 19:31:34.436378 12008 tablet_service.cc:1514] Processing DeleteTablet for tablet ce90a4411126470090149f9a40f77290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:31:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42286
I20260923 19:31:34.436688 11865 tablet_service.cc:1514] Processing DeleteTablet for tablet ce90a4411126470090149f9a40f77290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:31:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49584
I20260923 19:31:34.436712 12142 tablet_service.cc:1514] Processing DeleteTablet for tablet ce90a4411126470090149f9a40f77290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:31:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36582
I20260923 19:31:34.439286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 11797
I20260923 19:31:34.447628 12252 tablet_replica.cc:330] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: stopping tablet replica
I20260923 19:31:34.448354 12252 raft_consensus.cc:2229] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:31:34.449116 12252 raft_consensus.cc:2258] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:31:34.451748 12252 ts_tablet_manager.cc:1905] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:31:34.452478 12250 tablet_replica.cc:330] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: stopping tablet replica
I20260923 19:31:34.453022 12250 raft_consensus.cc:2229] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:31:34.454424 12251 tablet_replica.cc:330] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: stopping tablet replica
I20260923 19:31:34.454825 12251 raft_consensus.cc:2229] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:31:34.455268 12251 raft_consensus.cc:2258] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:31:34.459733 12251 ts_tablet_manager.cc:1905] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:31:34.461555 12252 ts_tablet_manager.cc:1918] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:31:34.461766 12252 log.cc:1196] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/ce90a4411126470090149f9a40f77290
Sep 23 19:31:34 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:31:34.463071 12250 raft_consensus.cc:2258] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:31:34.465873 12250 ts_tablet_manager.cc:1905] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:31:34.467218 12251 ts_tablet_manager.cc:1918] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:31:34.467381 12251 log.cc:1196] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/ce90a4411126470090149f9a40f77290
I20260923 19:31:34.467869 12251 ts_tablet_manager.cc:1939] T ce90a4411126470090149f9a40f77290 P 4c1cf27a24d34a35b51b3633d90fcf10: Deleting consensus metadata
I20260923 19:31:34.469805 11703 catalog_manager.cc:4841] TS 4c1cf27a24d34a35b51b3633d90fcf10 (127.0.97.1:38261): tablet ce90a4411126470090149f9a40f77290 (table test-table [id=00ef578cd0f949e6a8c4cbf2a7444459]) successfully deleted
I20260923 19:31:34.470238 12252 ts_tablet_manager.cc:1939] T ce90a4411126470090149f9a40f77290 P 4495af7d667544baa893e5026728c14d: Deleting consensus metadata
I20260923 19:31:34.472175 11701 catalog_manager.cc:4841] TS 4495af7d667544baa893e5026728c14d (127.0.97.3:33735): tablet ce90a4411126470090149f9a40f77290 (table test-table [id=00ef578cd0f949e6a8c4cbf2a7444459]) successfully deleted
I20260923 19:31:34.472280 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.440827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.0.97.1:38261)
0923 19:31:34.441038 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.441045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.445663 (+ 4618us) server_negotiation.cc:407] Connection header received
0923 19:31:34.445925 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.445931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.446026 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.446148 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:31:34.455627 (+ 9479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.456388 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.457749 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.458394 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.458529 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.464161 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:31:34.464176 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:31:34.464194 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:31:34.464250 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:31:34.468331 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.469311 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.469319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.469323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.469488 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.469996 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.470002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.470006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.470746 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:31:34.471024 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.471658 (+ 634us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.471910 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 19:31:34.473512 11925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:31:34.474977 12250 ts_tablet_manager.cc:1918] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:31:34.475270 12250 log.cc:1196] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/ce90a4411126470090149f9a40f77290
I20260923 19:31:34.475888 12250 ts_tablet_manager.cc:1939] T ce90a4411126470090149f9a40f77290 P 61c7041daf434135a0a7ca18885150d6: Deleting consensus metadata
I20260923 19:31:34.478017 11702 catalog_manager.cc:4841] TS 61c7041daf434135a0a7ca18885150d6 (127.0.97.2:41425): tablet ce90a4411126470090149f9a40f77290 (table test-table [id=00ef578cd0f949e6a8c4cbf2a7444459]) successfully deleted
I20260923 19:31:34.566974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 11797
I20260923 19:31:34.591957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 11936
Sep 23 19:31:34 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:31:34.610795 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.592796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.0.97.2:41425)
0923 19:31:34.592990 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.592996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.593092 (+ 96us) server_negotiation.cc:407] Connection header received
0923 19:31:34.593273 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.593278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.593351 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.593460 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:31:34.595383 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.596655 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.598639 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.599080 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.599182 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.604567 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:31:34.604580 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:31:34.604600 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:31:34.604636 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:31:34.607929 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.608657 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.608663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.608665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.608812 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.609180 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.609183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.609185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.609573 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:31:34.609878 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.610169 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.610423 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 19:31:34.611660 12068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:31:34.729444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 11936
I20260923 19:31:34.746605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 12079
Sep 23 19:31:34 dist-test-slave-kxfb krb5kdc[11241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:31:34.761794 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:31:34.747486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.0.97.3:33735)
0923 19:31:34.747690 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 19:31:34.747697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:31:34.747757 (+ 60us) server_negotiation.cc:407] Connection header received
0923 19:31:34.747968 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:31:34.747972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:31:34.748040 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:31:34.748122 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:31:34.749413 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.750146 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.751491 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:31:34.751873 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:31:34.751985 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:31:34.756419 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:31:34.756435 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:31:34.756453 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:31:34.756492 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:31:34.759116 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.759813 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.759820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.759822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.759933 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:31:34.760264 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:31:34.760268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:31:34.760271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:31:34.760701 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:31:34.760983 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 19:31:34.761249 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 19:31:34.761454 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 19:31:34.762938 12202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:31:34.853881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 12079
I20260923 19:31:34.869309 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 11685
I20260923 19:31:34.870579 11747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:31:34.995559 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 11685
I20260923 19:31:35.017345 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 19:31:35.353806 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 19:31:35.354043 388 mini_ranger.cc:66] Stopping Ranger...
I20260923 19:31:35.699975 388 mini_ranger.cc:68] Stopped Ranger
2026-09-23 19:31:35.700 UTC [11271] LOG: received smart shutdown request
2026-09-23 19:31:35.704 UTC [11271] LOG: background worker "logical replication launcher" (PID 11280) exited with exit code 1
2026-09-23 19:31:35.704 UTC [11275] LOG: shutting down
2026-09-23 19:31:35.875 UTC [11271] LOG: database system is shut down
2026-09-23T19:31:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49645 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:31:36 dist-test-slave-kxfb krb5kdc[12266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:31:36 dist-test-slave-kxfb krb5kdc[12266](info): set up 2 sockets
Sep 23 19:31:36 dist-test-slave-kxfb krb5kdc[12266](info): commencing operation
krb5kdc: starting...
W20260923 19:31:37.014881 388 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:31:37 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:31:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260923 19:31:37.369150 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhSGBcM/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/postgres -l logfile start
2026-09-23 19:31:40.040 UTC [12296] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 19:31:40.040 UTC [12296] LOG: listening on IPv4 address "127.0.97.20", port 41135
2026-09-23 19:31:40.046 UTC [12296] LOG: listening on Unix socket "/tmp/.s.PGSQL.41135"
2026-09-23 19:31:40.072 UTC [12299] LOG: database system was shut down at 2026-09-23 19:31:38 UTC
2026-09-23 19:31:40.076 UTC [12296] LOG: database system is ready to accept connections
I20260923 19:31:40.962353 388 mini_postgres.cc:96] Postgres bound to 41135
2026-09-23 19:31:40.981 UTC [12307] FATAL: database "slave" does not exist
127.0.97.20:41135 - accepting connections
I20260923 19:31:40.981807 388 mini_ranger.cc:161] Starting Ranger...
I20260923 19:31:41.010236 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 19:31:41.506382 388 mini_ranger.cc:87] Created ranger Postgres database
I20260923 19:31:41.506510 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-admin
2026-09-23 19:31:41,778 [I] DB FLAVOR :POSTGRES
2026-09-23 19:31:41,779 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 19:31:41,779 [I] Checking connection..
2026-09-23 19:31:41,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 19:31:42,043 [I] Checking connection passed.
2026-09-23 19:31:42,043 [I] --------- Verifying version history table ---------
2026-09-23 19:31:42,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 19:31:42,317 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 19:31:42,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 19:31:42,566 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 19:31:42,567 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 19:31:42,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 19:31:42.816 UTC [12371] WARNING: there is no transaction in progress
2026-09-23 19:31:42,823 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 19:31:42,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 19:31:43,091 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 19:31:43,092 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 19:31:43,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 19:31:43,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 19:31:43,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kxfb.c.gcp-upstream.internal','N') ;"
2026-09-23 19:31:43,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 19:31:44,142 [I] Table x_portal_user does not exist in database ranger
2026-09-23 19:31:44,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 19:31:44,414 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 19:31:44,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 19:31:44,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 19:31:44,926 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 19:31:44,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 19:31:45.938 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:45.963 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:45.980 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:45.997 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.014 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.033 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.105 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.112 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.127 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.137 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.148 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.161 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.171 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.178 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.186 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46.193 UTC [12479] WARNING: there is no transaction in progress
2026-09-23 19:31:46,878 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 19:31:46,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 19:31:47,136 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 19:31:47,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 19:31:47,407 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 19:31:47,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 19:31:47,654 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 19:31:47,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 19:31:47,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 19:31:48,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kxfb.c.gcp-upstream.internal';"
2026-09-23 19:31:48,409 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 19:31:48,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 19:31:48,684 [I] DB_PATCHES have already been applied
I20260923 19:31:48.691543 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhSGBcM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 19:31:48.691659 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260923 19:31:48.707818 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:51651
Sep 23, 2026 7:31:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 7:31:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 7:31:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 7:31:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 7:31:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 7:31:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 7:31:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 19:31:49.759018 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.777s user 0.000s sys 0.046s
Sep 23, 2026 7:32:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 7:32:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 23, 2026 7:32:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 7:32:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 7:32:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 19:32:12.712138 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 19:32:12 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191932, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 19:32:12.898247 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 19:32:12.921164 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 19:32:13.944615 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 19:32:13.944739 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-kms
I20260923 19:32:13.944787 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:41135
2026-09-23 19:32:14,277 [I] DB FLAVOR :POSTGRES
2026-09-23 19:32:14,277 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 19:32:14,277 [I] Checking connection
2026-09-23 19:32:14,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 19:32:14,529 [I] connection success
2026-09-23 19:32:14,529 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 19:32:14,529 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 19:32:14,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 19:32:14,781 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 19:32:14,782 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 19:32:14,782 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 19:32:14,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41135/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 19:32:15,055 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 19:32:15.598874 388 mini_ranger_kms.cc:310] Created kms service
I20260923 19:32:15.817901 388 mini_ranger_kms.cc:326] Created kudu user
I20260923 19:32:15.868580 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 19:32:15 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191932, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:16.311271 388 mini_ranger_kms.cc:384] Added ranger policy
I20260923 19:32:16.311687 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhSGBcM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhSGBcM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 19:32:16.311774 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260923 19:32:16.328816 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12660
I20260923 19:32:16.328975 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:58713
Sep 23, 2026 7:32:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 7:32:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 7:32:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 7:32:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 7:32:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 7:32:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 19:32:18.362107 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.464s user 0.060s sys 0.291s
I20260923 19:32:18.362397 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 19:32:18.362565 388 mini_ranger_kms.cc:401] 127.0.97.20:58713/kms/v1/keys
Sep 23 19:32:18 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790191938, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 19:32:19 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191938, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 19:32:19 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191938, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 19:32:21 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:32:21.932268 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37773
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37773
--ranger_config_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58713/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:22.268208 12710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:22.268571 12710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:22.268710 12710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:22.276602 12710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:32:22.276762 12710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:22.276854 12710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:22.276929 12710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:32:22.276996 12710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:32:22.288383 12710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:58713/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37773
--ranger_config_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:22.290076 12710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:22.291667 12710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:22.305058 12718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:22.304989 12716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:22.305019 12715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:22.305893 12710 server_base.cc:1034] running on GCE node
I20260923 19:32:22.306865 12710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:22.308599 12710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:22.309829 12710 hybrid_clock.cc:648] HybridClock initialized: now 1790191942309787 us; error 40 us; skew 500 ppm
I20260923 19:32:22.310256 12710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:22 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:22.318629 12710 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:32:22.320511 12710 webserver.cc:460] Webserver started at http://127.0.97.62:46809/ using document root <none> and password file <none>
I20260923 19:32:22.321348 12710 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:22.321478 12710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:22.321827 12710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 19:32:22 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:22.388568 12710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "80fb9d2350b34644907af6f281aa9006"
format_stamp: "Formatted at 2026-09-23 19:32:22 on dist-test-slave-kxfb"
server_key: "594217434466b74cc6e156bb5de3acc6"
server_key_iv: "c0db69516c630e132c9cb2ab946b3bad"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:22.389856 12710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "80fb9d2350b34644907af6f281aa9006"
format_stamp: "Formatted at 2026-09-23 19:32:22 on dist-test-slave-kxfb"
server_key: "594217434466b74cc6e156bb5de3acc6"
server_key_iv: "c0db69516c630e132c9cb2ab946b3bad"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:22.397821 12710 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260923 19:32:22.421211 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:22.423043 12710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:32:22.423262 12710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "80fb9d2350b34644907af6f281aa9006"
format_stamp: "Formatted at 2026-09-23 19:32:22 on dist-test-slave-kxfb"
server_key: "594217434466b74cc6e156bb5de3acc6"
server_key_iv: "c0db69516c630e132c9cb2ab946b3bad"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:22.423511 12710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:22.451212 12710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:22.455737 12710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:22.456077 12710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:22.475489 12710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37773
I20260923 19:32:22.475513 12776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37773 every 8 connection(s)
I20260923 19:32:22.477026 12710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:32:22.481346 12777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:22.482546 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 12710
I20260923 19:32:22.482789 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:32:22.483649 388 external_mini_cluster.cc:1428] Setting key 73683d696e4c9d66eccb7c9177c986ec
I20260923 19:32:22.497431 12777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Bootstrap starting.
Sep 23 19:32:22 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:22.501828 12777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:22.503520 12777 log.cc:824] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:22.507732 12777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: No bootstrap required, opened a new log
I20260923 19:32:22.509735 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:22.486189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.0.97.62:37773)
0923 19:32:22.487143 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:22.487157 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:22.487211 (+ 54us) server_negotiation.cc:407] Connection header received
0923 19:32:22.488957 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:22.488988 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:22.489532 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:22.489931 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:22.491731 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:22.493201 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:22.495172 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:22.495949 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:22.496100 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:22.500821 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:22.500835 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:22.500857 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:22.500912 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:22.504756 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:22.505633 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:22.505642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:22.505652 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:22.505886 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:22.506213 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:22.506217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:22.506220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:22.507017 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:22.507340 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:22.507736 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 19:32:22.508194 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":415,"threads_started":1}
I20260923 19:32:22.516415 12777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } }
I20260923 19:32:22.516825 12777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:22.516907 12777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fb9d2350b34644907af6f281aa9006, State: Initialized, Role: FOLLOWER
I20260923 19:32:22.517474 12777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } }
I20260923 19:32:22.517688 12777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:32:22.517831 12777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:32:22.518002 12777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:22.520646 12777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } }
I20260923 19:32:22.521234 12777 leader_election.cc:302] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80fb9d2350b34644907af6f281aa9006; no voters:
I20260923 19:32:22.521863 12777 leader_election.cc:288] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:32:22.521986 12782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:32:22.522706 12782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 1 LEADER]: Becoming Leader. State: Replica: 80fb9d2350b34644907af6f281aa9006, State: Running, Role: LEADER
I20260923 19:32:22.523427 12782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } }
I20260923 19:32:22.524343 12777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:32:22.526631 12783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80fb9d2350b34644907af6f281aa9006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } } }
I20260923 19:32:22.527112 12783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:22.526746 12784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80fb9d2350b34644907af6f281aa9006. Latest consensus state: current_term: 1 leader_uuid: "80fb9d2350b34644907af6f281aa9006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } } }
I20260923 19:32:22.527758 12784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:22.528898 12791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:32:22.534767 12791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:32:22.535416 12777 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 19:32:22.562866 12791 catalog_manager.cc:1347] Generated new cluster ID: b0c0da75fb2d4365b74a4ad517b618ce
I20260923 19:32:22.563066 12791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:32:22.585807 12791 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:32:22.588037 12791 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:32:22.597920 12791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Generated new TSK 0
I20260923 19:32:22.599084 12791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:32:23.382562 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37773
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:32:23 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790191943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 19:32:23.868525 12822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:23.869050 12822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:23.869323 12822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:23.881294 12822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:23.881471 12822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:23.881703 12822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:32:23.898181 12822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:23.900522 12822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:23.902737 12822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:23.922700 12833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:23.922686 12832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:23.930888 12835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:23.934902 12822 server_base.cc:1034] running on GCE node
I20260923 19:32:23.935870 12822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:23.942203 12822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:23.943744 12822 hybrid_clock.cc:648] HybridClock initialized: now 1790191943943651 us; error 89 us; skew 500 ppm
I20260923 19:32:23.944294 12822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:23 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:23.948767 12822 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:32:23.953879 12822 webserver.cc:460] Webserver started at http://127.0.97.1:41269/ using document root <none> and password file <none>
I20260923 19:32:23.955164 12822 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:23.955344 12822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:23.955780 12822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 19:32:23 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:23.987766 12822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "83fe43bb2bdc4254a9cf90d53cf433a2"
format_stamp: "Formatted at 2026-09-23 19:32:23 on dist-test-slave-kxfb"
server_key: "70b6365382408b114ef53c2d50d40137"
server_key_iv: "605e97684c14f36857f4f6d6b21d1398"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:23.989017 12822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "83fe43bb2bdc4254a9cf90d53cf433a2"
format_stamp: "Formatted at 2026-09-23 19:32:23 on dist-test-slave-kxfb"
server_key: "70b6365382408b114ef53c2d50d40137"
server_key_iv: "605e97684c14f36857f4f6d6b21d1398"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:23.997553 12822 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260923 19:32:24.048900 12842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:24.052057 12822 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.009s sys 0.008s
I20260923 19:32:24.052409 12822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "83fe43bb2bdc4254a9cf90d53cf433a2"
format_stamp: "Formatted at 2026-09-23 19:32:23 on dist-test-slave-kxfb"
server_key: "70b6365382408b114ef53c2d50d40137"
server_key_iv: "605e97684c14f36857f4f6d6b21d1398"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:24.052801 12822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:24.095276 12822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:24.103708 12822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:24.104110 12822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:24.105648 12822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:24.107781 12822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:24.107885 12822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:24.108019 12822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:24.108084 12822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:24.186149 12822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35877
I20260923 19:32:24.186177 12954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35877 every 8 connection(s)
I20260923 19:32:24.187628 12822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
Sep 23 19:32:24 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:24.194773 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 12822
I20260923 19:32:24.194993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:32:24.195766 388 external_mini_cluster.cc:1428] Setting key 5a9c1c79a86aa13b64df16077afe2b1d
Sep 23 19:32:24 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 19:32:24.242964 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:24.192633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37273 (local address 127.0.97.62:37773)
0923 19:32:24.219388 (+ 26755us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:24.219395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:24.219425 (+ 30us) server_negotiation.cc:407] Connection header received
0923 19:32:24.219510 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:24.219516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:24.219637 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:24.220016 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:24.222208 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:24.223361 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:24.226901 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:24.227573 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:24.227725 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:24.234782 (+ 7057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:24.234789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:24.234793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:24.234834 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:24.238130 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:24.239124 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:24.239129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:24.239132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:24.239277 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:24.239851 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:24.239856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:24.239860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:24.240239 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:24.240384 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:24.242210 (+ 1826us) server_negotiation.cc:299] Negotiation successful
0923 19:32:24.242489 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26574,"thread_start_us":27215,"threads_started":1}
I20260923 19:32:24.243216 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:24.191805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37773 (local address 127.0.97.1:37273)
0923 19:32:24.192748 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:24.192810 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:24.197571 (+ 4761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:24.220264 (+ 22693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:24.220280 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:24.221016 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:24.221951 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:24.221983 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:24.223696 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:24.223706 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:24.226642 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:24.226674 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:24.228129 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:24.228138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:24.228466 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:24.229288 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:24.229325 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:24.234475 (+ 5150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:24.238383 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:24.238393 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:24.238413 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:24.238980 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:24.239396 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:24.239436 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:24.239439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:24.239709 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:24.240458 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:24.240466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:24.240910 (+ 444us) client_negotiation.cc:769] Sending connection context
0923 19:32:24.241237 (+ 327us) client_negotiation.cc:240] Negotiation successful
0923 19:32:24.241775 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":855,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 19:32:24.245118 12955 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37773
I20260923 19:32:24.245673 12955 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:24.246889 12955 heartbeater.cc:511] Master 127.0.97.62:37773 requested a full tablet report, sending...
I20260923 19:32:24.249924 12742 ts_manager.cc:194] Registered new tserver with Master: 83fe43bb2bdc4254a9cf90d53cf433a2 (127.0.97.1:35877)
I20260923 19:32:24.251951 12742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:32:24.280936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37773
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:32:24 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260923 19:32:24.709585 12961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:24.710048 12961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:24.710244 12961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:24.724478 12961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:24.724697 12961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:24.725056 12961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:32:24.741269 12961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:24.743314 12961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:24.745014 12961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:24.763334 12968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:24.763365 12969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:24.769829 12971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:24.770670 12961 server_base.cc:1034] running on GCE node
I20260923 19:32:24.772269 12961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:24.781075 12961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:24.782936 12961 hybrid_clock.cc:648] HybridClock initialized: now 1790191944782809 us; error 116 us; skew 500 ppm
I20260923 19:32:24.783388 12961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:24 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:24.788795 12961 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:32:24.796058 12961 webserver.cc:460] Webserver started at http://127.0.97.2:45097/ using document root <none> and password file <none>
I20260923 19:32:24.797044 12961 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:24.797184 12961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:24.797537 12961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 19:32:24 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:24.815802 12961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "c94ea5fa4afa4317b6e15da5953e3513"
format_stamp: "Formatted at 2026-09-23 19:32:24 on dist-test-slave-kxfb"
server_key: "218830d86d013e7f28e385c53bc96542"
server_key_iv: "359ca5d03ed7f31f76cde9f73d4ff4a3"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:24.816663 12961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c94ea5fa4afa4317b6e15da5953e3513"
format_stamp: "Formatted at 2026-09-23 19:32:24 on dist-test-slave-kxfb"
server_key: "218830d86d013e7f28e385c53bc96542"
server_key_iv: "359ca5d03ed7f31f76cde9f73d4ff4a3"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:24.821796 12961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 19:32:24.838599 12985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:24.839973 12961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 19:32:24.840121 12961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "c94ea5fa4afa4317b6e15da5953e3513"
format_stamp: "Formatted at 2026-09-23 19:32:24 on dist-test-slave-kxfb"
server_key: "218830d86d013e7f28e385c53bc96542"
server_key_iv: "359ca5d03ed7f31f76cde9f73d4ff4a3"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:24.840341 12961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:24.852564 12961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:24.856598 12961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:24.856882 12961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:24.857903 12961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:24.859495 12961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:24.859573 12961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:24.859666 12961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:24.859728 12961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:24.899364 12961 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33173
I20260923 19:32:24.899408 13097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33173 every 8 connection(s)
I20260923 19:32:24.900589 12961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:32:24.908404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 12961
I20260923 19:32:24.908566 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:32:24.908970 388 external_mini_cluster.cc:1428] Setting key 0ba21af2472b145502c9afef11e34f68
Sep 23 19:32:24 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:24.926672 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:24.903405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38887 (local address 127.0.97.62:37773)
0923 19:32:24.904009 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:24.904017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:24.905062 (+ 1045us) server_negotiation.cc:407] Connection header received
0923 19:32:24.906622 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:24.906629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:24.906763 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:24.906945 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:24.908336 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:24.909617 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:24.911483 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:24.912154 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:24.912311 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:24.917069 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:24.917077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:24.917082 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:24.917126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:24.921134 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:24.922357 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:24.922363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:24.922367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:24.922525 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:24.923076 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:24.923081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:24.923085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:24.923558 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:24.923726 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:24.925806 (+ 2080us) server_negotiation.cc:299] Negotiation successful
0923 19:32:24.926204 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":270,"threads_started":1}
I20260923 19:32:24.927335 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:24.903765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37773 (local address 127.0.97.2:38887)
0923 19:32:24.904576 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:24.904632 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:24.906329 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:24.907100 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:24.907107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:24.907519 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:24.908110 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:24.908135 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:24.909832 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:24.909837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:24.911336 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:24.911351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:24.912434 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:24.912441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:24.912729 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:24.913559 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:24.913587 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:24.916726 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:24.921365 (+ 4639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:24.921378 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:24.921403 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:24.922176 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:24.922660 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:24.922665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:24.922669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:24.922944 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:24.923804 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:24.923814 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:24.924367 (+ 553us) client_negotiation.cc:769] Sending connection context
0923 19:32:24.924770 (+ 403us) client_negotiation.cc:240] Negotiation successful
0923 19:32:24.925461 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":340,"threads_started":1}
I20260923 19:32:24.929478 13098 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37773
I20260923 19:32:24.929876 13098 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:32:24.931056 13098 heartbeater.cc:511] Master 127.0.97.62:37773 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 19:32:24.933198 12742 ts_manager.cc:194] Registered new tserver with Master: c94ea5fa4afa4317b6e15da5953e3513 (127.0.97.2:33173)
I20260923 19:32:24.934370 12742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:32:24.959546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37773
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:32:25.255832 12955 heartbeater.cc:503] Master 127.0.97.62:37773 was elected leader, sending a full tablet report...
W20260923 19:32:25.293926 13104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:25.294224 13104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:25.294353 13104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:25.301470 13104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:25.301584 13104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:25.301695 13104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:32:25.311417 13104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:25.312661 13104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:25.313845 13104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:25.327629 13109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:25.327809 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:25.328042 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:25.329833 13104 server_base.cc:1034] running on GCE node
I20260923 19:32:25.330696 13104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:25.331929 13104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:25.333216 13104 hybrid_clock.cc:648] HybridClock initialized: now 1790191945333153 us; error 44 us; skew 500 ppm
I20260923 19:32:25.333592 13104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:25 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:25.337438 13104 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:32:25.339393 13104 webserver.cc:460] Webserver started at http://127.0.97.3:37787/ using document root <none> and password file <none>
I20260923 19:32:25.340216 13104 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:25.340340 13104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:25.340674 13104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 19:32:25 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:25.357620 13104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "55c276af6f04420aab17ed79c029b9af"
format_stamp: "Formatted at 2026-09-23 19:32:25 on dist-test-slave-kxfb"
server_key: "6afa5acebe88d4c65806e30d9a757f6e"
server_key_iv: "461914321ac34ef0eb7075c88502ca60"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:25.358536 13104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "55c276af6f04420aab17ed79c029b9af"
format_stamp: "Formatted at 2026-09-23 19:32:25 on dist-test-slave-kxfb"
server_key: "6afa5acebe88d4c65806e30d9a757f6e"
server_key_iv: "461914321ac34ef0eb7075c88502ca60"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:25.364547 13104 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 19:32:25.383018 13119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:25.384869 13104 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:32:25.385089 13104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "55c276af6f04420aab17ed79c029b9af"
format_stamp: "Formatted at 2026-09-23 19:32:25 on dist-test-slave-kxfb"
server_key: "6afa5acebe88d4c65806e30d9a757f6e"
server_key_iv: "461914321ac34ef0eb7075c88502ca60"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:25.385306 13104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:25.399690 13104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:25.403673 13104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:25.403961 13104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:25.405010 13104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:25.406529 13104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:25.406607 13104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:25.406708 13104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:25.406761 13104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:25.440876 13104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35159
I20260923 19:32:25.440953 13231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35159 every 8 connection(s)
I20260923 19:32:25.442147 13104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:32:25.449493 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 13104
I20260923 19:32:25.449664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:32:25.450124 388 external_mini_cluster.cc:1428] Setting key 40d070e494a2feec722cc927b05f5544
Sep 23 19:32:25 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:25.464102 13234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:25.445383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46359 (local address 127.0.97.62:37773)
0923 19:32:25.446138 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:25.446143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:25.447260 (+ 1117us) server_negotiation.cc:407] Connection header received
0923 19:32:25.449160 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:25.449166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:25.449288 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:25.449392 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:25.451151 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.452168 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.453928 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.454465 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.454572 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:25.458036 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:25.458042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:25.458046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:25.458088 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:25.461051 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:25.461798 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:25.461802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:25.461805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:25.461943 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:25.462357 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:25.462360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:25.462362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:25.462674 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:25.462810 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:25.463551 (+ 741us) server_negotiation.cc:299] Negotiation successful
0923 19:32:25.463770 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":358,"threads_started":1}
I20260923 19:32:25.465245 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:25.445951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37773 (local address 127.0.97.3:46359)
0923 19:32:25.447040 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:25.447099 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:25.448835 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:25.449710 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:25.449724 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:25.450201 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:25.450902 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:25.450926 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.452314 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.452319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:25.453781 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:25.453794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.454674 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.454679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:25.454913 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:25.455631 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:25.455653 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:25.457778 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:25.461226 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:25.461234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:25.461248 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:25.461692 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:25.462045 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:25.462048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:25.462050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:25.462228 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:25.462831 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:25.462838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:25.463175 (+ 337us) client_negotiation.cc:769] Sending connection context
0923 19:32:25.463422 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 19:32:25.463903 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":528,"threads_started":1}
I20260923 19:32:25.467026 13232 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37773
I20260923 19:32:25.467305 13232 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:25.467998 13232 heartbeater.cc:511] Master 127.0.97.62:37773 requested a full tablet report, sending...
I20260923 19:32:25.469556 12741 ts_manager.cc:194] Registered new tserver with Master: 55c276af6f04420aab17ed79c029b9af (127.0.97.3:35159)
I20260923 19:32:25.470567 12741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46359
I20260923 19:32:25.479156 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:32:25.481839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 12822
Sep 23 19:32:25 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:32:25.499963 13237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:25.483001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.0.97.1:35877)
0923 19:32:25.483751 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:25.483758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:25.483783 (+ 25us) server_negotiation.cc:407] Connection header received
0923 19:32:25.483843 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:25.483849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:25.484116 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:25.484257 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:25.485715 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.486801 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.488265 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.488819 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.488951 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:25.493739 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:25.493759 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:25.493780 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:25.493834 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:25.496973 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:25.497721 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:25.497730 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:25.497732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:25.497927 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:25.498357 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:25.498360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:25.498362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:25.498782 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:25.499054 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:25.499380 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 19:32:25.499630 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":373,"threads_started":1}
I20260923 19:32:25.501564 12950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:25.591768 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 12822
I20260923 19:32:25.606992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 12961
Sep 23 19:32:25 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:32:25.624228 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:25.608124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.0.97.2:33173)
0923 19:32:25.608782 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:25.608790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:25.608824 (+ 34us) server_negotiation.cc:407] Connection header received
0923 19:32:25.608894 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:25.608901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:25.609122 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:25.609280 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:25.610635 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.611639 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.613037 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.613576 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.613678 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:25.618076 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:25.618089 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:25.618104 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:25.618149 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:25.621113 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:25.621916 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:25.621925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:25.621927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:25.622087 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:25.622557 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:25.622561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:25.622563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:25.622948 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:25.623202 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:25.623567 (+ 365us) server_negotiation.cc:299] Negotiation successful
0923 19:32:25.623851 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":329,"threads_started":1}
I20260923 19:32:25.625707 13093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:25.708052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 12961
I20260923 19:32:25.726826 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 13104
Sep 23 19:32:25 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:32:25.744577 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:25.727971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.0.97.3:35159)
0923 19:32:25.728786 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:25.728816 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:25.728851 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:32:25.728924 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:25.728931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:25.729206 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:25.729373 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:25.730852 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.731931 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.733347 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:25.733960 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:25.734076 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:25.738593 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:25.738605 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:25.738623 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:25.738660 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:25.741582 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:25.742375 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:25.742383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:25.742386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:25.742529 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:25.742881 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:25.742885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:25.742887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:25.743277 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:25.743609 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:25.743893 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 19:32:25.744173 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":454,"threads_started":1}
I20260923 19:32:25.746198 13227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:25.837198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 13104
I20260923 19:32:25.851574 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 12710
I20260923 19:32:25.853205 12772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:25.963729 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 12710
I20260923 19:32:25.986234 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37773
--webserver_interface=127.0.97.62
--webserver_port=46809
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37773
--ranger_config_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58713/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58713/kms with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:26.304765 13244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:26.305078 13244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:26.305166 13244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:26.312410 13244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:32:26.312530 13244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:26.312575 13244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:26.312635 13244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:32:26.312685 13244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:32:26.322674 13244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58713/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37773
--ranger_config_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=46809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:26.323920 13244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:26.325143 13244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:26.337594 13250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:26.337643 13252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:26.337643 13249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:26.338742 13244 server_base.cc:1034] running on GCE node
I20260923 19:32:26.339589 13244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:26.341041 13244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:26.342319 13244 hybrid_clock.cc:648] HybridClock initialized: now 1790191946342220 us; error 57 us; skew 500 ppm
I20260923 19:32:26.342706 13244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:26 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:26.345809 13244 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:32:26.347420 13244 webserver.cc:460] Webserver started at http://127.0.97.62:46809/ using document root <none> and password file <none>
I20260923 19:32:26.348069 13244 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:26.348176 13244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 19:32:26 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:26.395140 13244 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:32:26.398500 13260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:26.400172 13244 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:32:26.400389 13244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "80fb9d2350b34644907af6f281aa9006"
format_stamp: "Formatted at 2026-09-23 19:32:22 on dist-test-slave-kxfb"
server_key: "594217434466b74cc6e156bb5de3acc6"
server_key_iv: "c0db69516c630e132c9cb2ab946b3bad"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:26.401366 13244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:26.415583 13244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:26.419593 13244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:26.419883 13244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:26.442819 13244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37773
I20260923 19:32:26.442869 13311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37773 every 8 connection(s)
I20260923 19:32:26.444332 13244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:32:26.452023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 13244
I20260923 19:32:26.452725 13312 sys_catalog.cc:263] Verifying existing consensus state
I20260923 19:32:26.453199 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35877
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37773
--webserver_port=41269
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58713/kms with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:32:26.457752 13312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Bootstrap starting.
I20260923 19:32:26.480913 13312 log.cc:824] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:26.491544 13312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 19:32:26.492230 13312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Bootstrap complete.
I20260923 19:32:26.501952 13312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } }
I20260923 19:32:26.503075 13312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fb9d2350b34644907af6f281aa9006, State: Initialized, Role: FOLLOWER
I20260923 19:32:26.503866 13312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } }
I20260923 19:32:26.504124 13312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:32:26.504245 13312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:32:26.504424 13312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 1 FOLLOWER]: Advancing to term 2
I20260923 19:32:26.509323 13312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } }
I20260923 19:32:26.509981 13312 leader_election.cc:302] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80fb9d2350b34644907af6f281aa9006; no voters:
I20260923 19:32:26.510700 13312 leader_election.cc:288] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 19:32:26.510885 13316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 19:32:26.512101 13316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [term 2 LEADER]: Becoming Leader. State: Replica: 80fb9d2350b34644907af6f281aa9006, State: Running, Role: LEADER
I20260923 19:32:26.512919 13316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } }
I20260923 19:32:26.513397 13312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:32:26.516326 13318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80fb9d2350b34644907af6f281aa9006. Latest consensus state: current_term: 2 leader_uuid: "80fb9d2350b34644907af6f281aa9006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } } }
I20260923 19:32:26.516794 13318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:26.517408 13317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "80fb9d2350b34644907af6f281aa9006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fb9d2350b34644907af6f281aa9006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37773 } } }
I20260923 19:32:26.517714 13317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:26.530094 13323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:32:26.536463 13323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:32:26.538033 13323 catalog_manager.cc:1259] Loaded cluster ID: b0c0da75fb2d4365b74a4ad517b618ce
I20260923 19:32:26.538246 13323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:32:26.542505 13323 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:32:26.544202 13323 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Loaded TSK: 0
I20260923 19:32:26.545902 13323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:32:26.548935 13312 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260923 19:32:26.871444 13314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:26.871932 13314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:26.872124 13314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:26.883646 13314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:26.883865 13314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:26.884092 13314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:32:26.899048 13314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41269
--tserver_master_addrs=127.0.97.62:37773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:26.900411 13314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:26.902020 13314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:26.941115 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:26.943200 13350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:26.948277 13314 server_base.cc:1034] running on GCE node
W20260923 19:32:26.948751 13352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:26.949960 13314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:26.951898 13314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:26.953949 13314 hybrid_clock.cc:648] HybridClock initialized: now 1790191946953440 us; error 498 us; skew 500 ppm
I20260923 19:32:26.954406 13314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:26 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:26.957593 13314 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:32:26.959839 13314 webserver.cc:460] Webserver started at http://127.0.97.1:41269/ using document root <none> and password file <none>
I20260923 19:32:26.960737 13314 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:26.960886 13314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 19:32:26 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:26.986595 13314 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:32:26.989655 13360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:26.991271 13314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:32:26.991441 13314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "83fe43bb2bdc4254a9cf90d53cf433a2"
format_stamp: "Formatted at 2026-09-23 19:32:23 on dist-test-slave-kxfb"
server_key: "70b6365382408b114ef53c2d50d40137"
server_key_iv: "605e97684c14f36857f4f6d6b21d1398"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:26.992399 13314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:27.007395 13314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:27.011538 13314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:27.011806 13314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:27.012822 13314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:27.014374 13314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:27.014482 13314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:27.014591 13314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:27.014664 13314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:27.129359 13314 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35877
I20260923 19:32:27.129395 13472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35877 every 8 connection(s)
I20260923 19:32:27.131353 13314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:32:27.142000 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 13314
I20260923 19:32:27.142966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33173
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37773
--webserver_port=45097
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58713/kms with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:32:27 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:27.173583 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:27.136148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37773 (local address 127.0.97.1:38529)
0923 19:32:27.145807 (+ 9659us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:27.145868 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:27.148566 (+ 2698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:27.152265 (+ 3699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:27.152277 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:27.152908 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:27.153641 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:27.153666 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:27.156110 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:27.156117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:27.158101 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:27.158132 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:27.159908 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:27.159963 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:27.160225 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:27.161012 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:27.161040 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:27.163843 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:27.168293 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:27.168303 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:27.168319 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:27.168913 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:27.169493 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:27.169497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:27.169500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:27.169730 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:27.170892 (+ 1162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:27.170900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:27.171300 (+ 400us) client_negotiation.cc:769] Sending connection context
0923 19:32:27.171614 (+ 314us) client_negotiation.cc:240] Negotiation successful
0923 19:32:27.172194 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9313,"thread_start_us":9015,"threads_started":1}
I20260923 19:32:27.177340 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:27.137666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38529 (local address 127.0.97.62:37773)
0923 19:32:27.150863 (+ 13197us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:27.150878 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:27.150923 (+ 45us) server_negotiation.cc:407] Connection header received
0923 19:32:27.151288 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:27.151301 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:27.151687 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:27.152069 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:27.153893 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:27.155910 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:27.158347 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:27.159354 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:27.159531 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:27.164189 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:27.164200 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:27.164220 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:27.164265 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:27.168099 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:27.169134 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:27.169143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:27.169149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:27.169346 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:27.169886 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:27.169891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:27.169894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:27.170686 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:27.175416 (+ 4730us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:27.175689 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 19:32:27.176110 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12920,"thread_start_us":12645,"threads_started":1}
I20260923 19:32:27.181967 13473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37773
I20260923 19:32:27.182484 13473 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:27.183498 13473 heartbeater.cc:511] Master 127.0.97.62:37773 requested a full tablet report, sending...
W20260923 19:32:27.187335 13473 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37773 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 19:32:27.187572 13473 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:27.187896 13473 heartbeater.cc:511] Master 127.0.97.62:37773 requested a full tablet report, sending...
I20260923 19:32:27.189348 13473 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:27.189723 13473 heartbeater.cc:511] Master 127.0.97.62:37773 requested a full tablet report, sending...
W20260923 19:32:27.190932 13473 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 19:32:27.551817 13477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:27.552311 13477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:27.552516 13477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:27.561029 13477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:27.561141 13477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:27.561295 13477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:32:27.572101 13477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45097
--tserver_master_addrs=127.0.97.62:37773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:27.573510 13477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:27.574853 13477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:27.590613 13491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:27.590656 13490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:27.592447 13493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:27.593196 13477 server_base.cc:1034] running on GCE node
I20260923 19:32:27.593799 13477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:27.594811 13477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:27.596077 13477 hybrid_clock.cc:648] HybridClock initialized: now 1790191947596014 us; error 44 us; skew 500 ppm
I20260923 19:32:27.596422 13477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:27 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:27.602759 13477 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:32:27.604333 13477 webserver.cc:460] Webserver started at http://127.0.97.2:45097/ using document root <none> and password file <none>
I20260923 19:32:27.605082 13477 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:27.605242 13477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 19:32:27 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:27.635035 13477 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.009s sys 0.000s
I20260923 19:32:27.639278 13501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:27.641119 13477 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:32:27.641305 13477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "c94ea5fa4afa4317b6e15da5953e3513"
format_stamp: "Formatted at 2026-09-23 19:32:24 on dist-test-slave-kxfb"
server_key: "218830d86d013e7f28e385c53bc96542"
server_key_iv: "359ca5d03ed7f31f76cde9f73d4ff4a3"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:27.642426 13477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:27.655447 13477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:27.659432 13477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:27.659703 13477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:27.660698 13477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:27.662120 13477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:27.662262 13477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:27.662400 13477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:27.662470 13477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:27.725028 13477 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33173
I20260923 19:32:27.725129 13615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33173 every 8 connection(s)
I20260923 19:32:27.727165 13477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:32:27.730654 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 13477
I20260923 19:32:27.731501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35159
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37773
--webserver_port=37787
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58713/kms with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:32:27 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:27.783959 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:27.730339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59261 (local address 127.0.97.62:37773)
0923 19:32:27.745197 (+ 14858us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:27.745204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:27.761199 (+ 15995us) server_negotiation.cc:407] Connection header received
0923 19:32:27.764184 (+ 2985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:27.764191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:27.764294 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:27.764390 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:27.766934 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:27.768202 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:27.770841 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:27.771439 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:27.771567 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:27.775817 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:27.775823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:27.775828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:27.775867 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:27.779257 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:27.780284 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:27.780292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:27.780296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:27.780567 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:27.781357 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:27.781362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:27.781365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:27.781842 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:27.782020 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:27.783243 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0923 19:32:27.783541 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14700,"thread_start_us":11977,"threads_started":1}
I20260923 19:32:27.785357 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:27.731113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37773 (local address 127.0.97.2:59261)
0923 19:32:27.760962 (+ 29849us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:27.761036 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:27.763777 (+ 2741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:27.765007 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:27.765020 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:27.765673 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:27.766695 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:27.766721 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:27.768434 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:27.768440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:27.770653 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:27.770678 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:27.771933 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:27.771941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:27.772210 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:27.772976 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:27.773004 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:27.775570 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:27.779435 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:27.779446 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:27.779466 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:27.780009 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:27.780783 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:27.780789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:27.780793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:27.781175 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:27.782144 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:27.782155 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:27.782745 (+ 590us) client_negotiation.cc:769] Sending connection context
0923 19:32:27.783099 (+ 354us) client_negotiation.cc:240] Negotiation successful
0923 19:32:27.783780 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29396,"thread_start_us":29130,"threads_started":1}
I20260923 19:32:27.787457 13616 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37773
I20260923 19:32:27.787853 13616 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:27.788913 13616 heartbeater.cc:511] Master 127.0.97.62:37773 requested a full tablet report, sending...
I20260923 19:32:27.791633 13277 ts_manager.cc:194] Registered new tserver with Master: c94ea5fa4afa4317b6e15da5953e3513 (127.0.97.2:33173)
I20260923 19:32:27.794014 13277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59261
Sep 23 19:32:27 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790191947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 19:32:28.146798 13620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:28.147109 13620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:28.147265 13620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:28.157228 13620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:28.157375 13620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:28.157532 13620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:32:28.172261 13620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37787
--tserver_master_addrs=127.0.97.62:37773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:28.174225 13620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:28.176208 13620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 19:32:28.191082 13473 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:28.191555 13473 heartbeater.cc:511] Master 127.0.97.62:37773 requested a full tablet report, sending...
I20260923 19:32:28.192948 13277 ts_manager.cc:194] Registered new tserver with Master: 83fe43bb2bdc4254a9cf90d53cf433a2 (127.0.97.1:35877)
I20260923 19:32:28.194123 13277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38529
W20260923 19:32:28.198105 13629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:28.198151 13628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:28.210006 13620 server_base.cc:1034] running on GCE node
W20260923 19:32:28.211298 13631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:28.212366 13620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:28.213852 13620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:28.215267 13620 hybrid_clock.cc:648] HybridClock initialized: now 1790191948215171 us; error 82 us; skew 500 ppm
I20260923 19:32:28.215694 13620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:28 dist-test-slave-kxfb krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:28.220485 13620 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:32:28.226699 13620 webserver.cc:460] Webserver started at http://127.0.97.3:37787/ using document root <none> and password file <none>
I20260923 19:32:28.227742 13620 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:28.227946 13620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 19:32:28 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 19:32:28.257014 13620 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260923 19:32:28.261286 13639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:28.263537 13620 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260923 19:32:28.263783 13620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "55c276af6f04420aab17ed79c029b9af"
format_stamp: "Formatted at 2026-09-23 19:32:25 on dist-test-slave-kxfb"
server_key: "6afa5acebe88d4c65806e30d9a757f6e"
server_key_iv: "461914321ac34ef0eb7075c88502ca60"
server_key_version: "kuduclusterkey@0"
I20260923 19:32:28.265425 13620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:28.285017 13620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:28.291309 13620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:28.291791 13620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:28.293290 13620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:28.295440 13620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:28.295598 13620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 19:32:28.295756 13620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:28.295854 13620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:28.373075 13620 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35159
I20260923 19:32:28.373103 13751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35159 every 8 connection(s)
I20260923 19:32:28.375105 13620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:32:28.383075 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 13620
Sep 23 19:32:28 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 19:32:28 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:28.406836 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.384062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42134 (local address 127.0.97.62:37773)
0923 19:32:28.385563 (+ 1501us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.385569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.386436 (+ 867us) server_negotiation.cc:407] Connection header received
0923 19:32:28.386517 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.386523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.386611 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.386725 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:28.390362 (+ 3637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.391559 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.393511 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.394136 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.394252 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.398271 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:28.398278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:28.398282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:28.398352 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:28.402737 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:28.403625 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:28.403630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:28.403634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:28.403803 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:28.404225 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:28.404230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:28.404233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:28.404597 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:28.404863 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:28.406120 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0923 19:32:28.406396 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":1231,"threads_started":1}
I20260923 19:32:28.407176 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.378483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52283 (local address 127.0.97.62:37773)
0923 19:32:28.382731 (+ 4248us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.382740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.385149 (+ 2409us) server_negotiation.cc:407] Connection header received
0923 19:32:28.387533 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.387538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.387608 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.387681 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:28.390107 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.391447 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.394792 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.395409 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.395497 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.400096 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:28.400103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:28.400107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:28.400161 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:28.403558 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:28.404459 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:28.404463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:28.404466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:28.404575 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:28.405077 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:28.405083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:28.405086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:28.405426 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:28.405534 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:28.406618 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0923 19:32:28.406816 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4058,"thread_start_us":3838,"threads_started":1}
I20260923 19:32:28.409813 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.379301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37773 (local address 127.0.97.3:52283)
0923 19:32:28.384570 (+ 5269us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:28.384627 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:28.387218 (+ 2591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:28.388313 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:28.388327 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:28.388915 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:28.389871 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.389904 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.391648 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.391653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.394609 (+ 2956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.394650 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.395864 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.395870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.396177 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.396966 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:28.396999 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:28.399783 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:28.403700 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:28.403710 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:28.403732 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:28.404341 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:28.404707 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:28.404711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:28.404714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:28.404953 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:28.405633 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:28.405642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:28.406137 (+ 495us) client_negotiation.cc:769] Sending connection context
0923 19:32:28.407247 (+ 1110us) client_negotiation.cc:240] Negotiation successful
0923 19:32:28.408269 (+ 1022us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4949,"thread_start_us":4687,"threads_started":1}
I20260923 19:32:28.415208 13752 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37773
I20260923 19:32:28.415596 13752 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:28.416626 13752 heartbeater.cc:511] Master 127.0.97.62:37773 requested a full tablet report, sending...
I20260923 19:32:28.418783 13276 ts_manager.cc:194] Registered new tserver with Master: 55c276af6f04420aab17ed79c029b9af (127.0.97.3:35159)
I20260923 19:32:28.420275 13276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52283
I20260923 19:32:28.423916 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:32:28.471294 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.458474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.0.97.62:37773)
0923 19:32:28.458702 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.458709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.459212 (+ 503us) server_negotiation.cc:407] Connection header received
0923 19:32:28.459392 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.459397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.459485 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.459569 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:28.461017 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.461742 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.463145 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.463575 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.463691 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.466160 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:28.466166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:28.466169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:28.466201 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:28.468725 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:28.469353 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:28.469359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:28.469361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:28.469477 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:28.469812 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:28.469817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:28.469819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:28.470102 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:28.470375 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:28.470672 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 19:32:28.470971 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 19:32:28.476366 13277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:32:28.483894 13277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:32:28.553771 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.505172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35877 (local address 127.0.0.1:49100)
0923 19:32:28.541274 (+ 36102us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:28.541298 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:28.541426 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:28.542591 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:28.542599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:28.542630 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:28.543039 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.543052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.544987 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.544992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.549345 (+ 4353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.549473 (+ 128us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.552689 (+ 3216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.552697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.552942 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.552999 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 19:32:28.553133 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 19:32:28.553378 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35975,"thread_start_us":35459,"threads_started":1}
I20260923 19:32:28.554879 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.505855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.0.97.1:35877)
0923 19:32:28.540419 (+ 34564us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.540429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.541553 (+ 1124us) server_negotiation.cc:407] Connection header received
0923 19:32:28.541676 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.541685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.542012 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.542240 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:28.543257 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.544829 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.550268 (+ 5439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.552043 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.552184 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.552481 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:28.554338 (+ 1857us) server_negotiation.cc:299] Negotiation successful
0923 19:32:28.554546 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34371,"thread_start_us":34581,"threads_started":1}
I20260923 19:32:28.561311 13408 tablet_service.cc:1467] Processing CreateTablet for tablet 9ceb2acfd1534202a9f26f8701d22560 (DEFAULT_TABLE table=test-table [id=8f2176a61ddb426bb9bfe2a45f3af6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:32:28.563326 13408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ceb2acfd1534202a9f26f8701d22560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:28.563800 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.500633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33173 (local address 127.0.0.1:48770)
0923 19:32:28.539645 (+ 39012us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:28.539672 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:28.539853 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:28.549613 (+ 9760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:28.549619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:28.549649 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:28.549961 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.549974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.554978 (+ 5004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.554985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.557048 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.557064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.562952 (+ 5888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.562958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.563143 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.563182 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 19:32:28.563254 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 19:32:28.563406 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38831,"thread_start_us":40380,"threads_started":1}
I20260923 19:32:28.565390 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.501313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.0.97.2:33173)
0923 19:32:28.548257 (+ 46944us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.548266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.548299 (+ 33us) server_negotiation.cc:407] Connection header received
0923 19:32:28.548403 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.548411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.548666 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.550220 (+ 1554us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:28.550496 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.552006 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.557297 (+ 5291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.558824 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.564586 (+ 5762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.564809 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:28.564897 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 19:32:28.565077 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46766,"thread_start_us":46643,"threads_started":1}
I20260923 19:32:28.566581 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.499107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35159 (local address 127.0.0.1:37666)
0923 19:32:28.517164 (+ 18057us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:28.517232 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:28.517465 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:28.552210 (+ 34745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:28.552215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:28.552244 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:28.552660 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.552675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.558989 (+ 6314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.558995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.560956 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.560972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.565782 (+ 4810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.565789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.565993 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.566029 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 19:32:28.566111 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 19:32:28.566245 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17622,"thread_start_us":17466,"threads_started":1}
I20260923 19:32:28.567939 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.499745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.97.3:35159)
0923 19:32:28.549031 (+ 49286us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.549039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.549069 (+ 30us) server_negotiation.cc:407] Connection header received
0923 19:32:28.549177 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.549184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.549380 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.552871 (+ 3491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:28.553205 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.554789 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.561184 (+ 6395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.562685 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.567105 (+ 4420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.567349 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:28.567422 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 19:32:28.567599 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48047,"thread_start_us":48587,"threads_started":1}
I20260923 19:32:28.572667 13551 tablet_service.cc:1467] Processing CreateTablet for tablet 9ceb2acfd1534202a9f26f8701d22560 (DEFAULT_TABLE table=test-table [id=8f2176a61ddb426bb9bfe2a45f3af6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:32:28.574532 13551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ceb2acfd1534202a9f26f8701d22560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:28.574383 13687 tablet_service.cc:1467] Processing CreateTablet for tablet 9ceb2acfd1534202a9f26f8701d22560 (DEFAULT_TABLE table=test-table [id=8f2176a61ddb426bb9bfe2a45f3af6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:32:28.576152 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ceb2acfd1534202a9f26f8701d22560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:28.598022 13773 tablet_bootstrap.cc:492] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: Bootstrap starting.
I20260923 19:32:28.604876 13773 tablet_bootstrap.cc:654] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:28.606721 13773 log.cc:824] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:28.615161 13774 tablet_bootstrap.cc:492] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: Bootstrap starting.
I20260923 19:32:28.616065 13775 tablet_bootstrap.cc:492] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2: Bootstrap starting.
I20260923 19:32:28.616966 13773 tablet_bootstrap.cc:492] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: No bootstrap required, opened a new log
I20260923 19:32:28.618132 13773 ts_tablet_manager.cc:1397] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.011s
I20260923 19:32:28.620127 13774 tablet_bootstrap.cc:654] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:28.620332 13775 tablet_bootstrap.cc:654] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:28.621600 13774 log.cc:824] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:28.621593 13775 log.cc:824] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:28.633250 13775 tablet_bootstrap.cc:492] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2: No bootstrap required, opened a new log
I20260923 19:32:28.633751 13775 ts_tablet_manager.cc:1397] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260923 19:32:28.629441 13773 raft_consensus.cc:348] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } }
I20260923 19:32:28.639196 13773 raft_consensus.cc:374] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:28.639400 13773 raft_consensus.cc:729] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94ea5fa4afa4317b6e15da5953e3513, State: Initialized, Role: FOLLOWER
I20260923 19:32:28.641856 13774 tablet_bootstrap.cc:492] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: No bootstrap required, opened a new log
I20260923 19:32:28.642841 13773 consensus_queue.cc:260] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } }
I20260923 19:32:28.643831 13774 ts_tablet_manager.cc:1397] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.017s
I20260923 19:32:28.645328 13775 raft_consensus.cc:348] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } }
I20260923 19:32:28.646061 13775 raft_consensus.cc:374] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:28.646201 13775 raft_consensus.cc:729] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fe43bb2bdc4254a9cf90d53cf433a2, State: Initialized, Role: FOLLOWER
I20260923 19:32:28.647161 13775 consensus_queue.cc:260] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } }
I20260923 19:32:28.648764 13616 heartbeater.cc:503] Master 127.0.97.62:37773 was elected leader, sending a full tablet report...
I20260923 19:32:28.652300 13473 heartbeater.cc:503] Master 127.0.97.62:37773 was elected leader, sending a full tablet report...
I20260923 19:32:28.653405 13775 ts_tablet_manager.cc:1428] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2: Time spent starting tablet: real 0.020s user 0.010s sys 0.005s
I20260923 19:32:28.654680 13773 ts_tablet_manager.cc:1428] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: Time spent starting tablet: real 0.036s user 0.014s sys 0.000s
I20260923 19:32:28.662858 13774 raft_consensus.cc:348] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } }
I20260923 19:32:28.663421 13774 raft_consensus.cc:374] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:28.663554 13774 raft_consensus.cc:729] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c276af6f04420aab17ed79c029b9af, State: Initialized, Role: FOLLOWER
I20260923 19:32:28.664386 13774 consensus_queue.cc:260] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } }
I20260923 19:32:28.671695 13752 heartbeater.cc:503] Master 127.0.97.62:37773 was elected leader, sending a full tablet report...
I20260923 19:32:28.672529 13774 ts_tablet_manager.cc:1428] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: Time spent starting tablet: real 0.029s user 0.016s sys 0.003s
Sep 23 19:32:28 dist-test-slave-kxfb krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260923 19:32:28.729813 13617 tablet.cc:2369] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:32:28.777592 13780 raft_consensus.cc:482] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:32:28.778015 13780 raft_consensus.cc:504] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } }
I20260923 19:32:28.782471 13780 leader_election.cc:288] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55c276af6f04420aab17ed79c029b9af (127.0.97.3:35159), c94ea5fa4afa4317b6e15da5953e3513 (127.0.97.2:33173)
I20260923 19:32:28.798735 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.782037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37117 (local address 127.0.97.3:35159)
0923 19:32:28.782558 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.782564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.786075 (+ 3511us) server_negotiation.cc:407] Connection header received
0923 19:32:28.787243 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.787249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.787331 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.787398 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:28.788176 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.789266 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.796319 (+ 7053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.797521 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.797627 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.797752 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:28.798337 (+ 585us) server_negotiation.cc:299] Negotiation successful
0923 19:32:28.798468 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260923 19:32:28.799759 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.781309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35159 (local address 127.0.97.1:37117)
0923 19:32:28.785890 (+ 4581us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:28.785934 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:28.787013 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:28.787613 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:28.787618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:28.787647 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:28.787994 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.788008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.793942 (+ 5934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.793947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.796126 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.796143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.797943 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.797949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.798131 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.798166 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 19:32:28.799245 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0923 19:32:28.799401 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4392,"threads_started":1}
I20260923 19:32:28.807415 13707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ceb2acfd1534202a9f26f8701d22560" candidate_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c276af6f04420aab17ed79c029b9af" is_pre_election: true
I20260923 19:32:28.808032 13707 raft_consensus.cc:2454] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83fe43bb2bdc4254a9cf90d53cf433a2 in term 0.
I20260923 19:32:28.809029 13364 leader_election.cc:302] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55c276af6f04420aab17ed79c029b9af, 83fe43bb2bdc4254a9cf90d53cf433a2; no voters:
I20260923 19:32:28.811303 13780 raft_consensus.cc:2781] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:32:28.811444 13780 raft_consensus.cc:482] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:32:28.811538 13780 raft_consensus.cc:3037] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:28.814827 13780 raft_consensus.cc:504] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } }
I20260923 19:32:28.815644 13780 leader_election.cc:288] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [CANDIDATE]: Term 1 election: Requested vote from peers 55c276af6f04420aab17ed79c029b9af (127.0.97.3:35159), c94ea5fa4afa4317b6e15da5953e3513 (127.0.97.2:33173)
I20260923 19:32:28.816416 13707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ceb2acfd1534202a9f26f8701d22560" candidate_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c276af6f04420aab17ed79c029b9af"
I20260923 19:32:28.816620 13707 raft_consensus.cc:3037] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:28.819813 13707 raft_consensus.cc:2454] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83fe43bb2bdc4254a9cf90d53cf433a2 in term 1.
I20260923 19:32:28.821682 13364 leader_election.cc:302] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55c276af6f04420aab17ed79c029b9af, 83fe43bb2bdc4254a9cf90d53cf433a2; no voters:
I20260923 19:32:28.822042 13780 raft_consensus.cc:2781] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:32:28.824026 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.782112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37405 (local address 127.0.97.2:33173)
0923 19:32:28.782467 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.782475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.801190 (+ 18715us) server_negotiation.cc:407] Connection header received
0923 19:32:28.801768 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.801775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.801871 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.801934 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:28.805445 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.806492 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.809143 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.810410 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.810589 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.810698 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:28.823516 (+ 12818us) server_negotiation.cc:299] Negotiation successful
0923 19:32:28.823688 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260923 19:32:28.825531 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.781331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33173 (local address 127.0.97.1:37405)
0923 19:32:28.801060 (+ 19729us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:28.801080 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:28.801530 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:28.802146 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:28.802151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:28.802178 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:28.802738 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.802751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.807016 (+ 4265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.807022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.808985 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:28.809002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.822613 (+ 13611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.822621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:28.822827 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.822867 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 19:32:28.824433 (+ 1566us) client_negotiation.cc:240] Negotiation successful
0923 19:32:28.824594 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19581,"mutex_wait_us":1,"thread_start_us":4098,"threads_started":1}
I20260923 19:32:28.827616 13571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ceb2acfd1534202a9f26f8701d22560" candidate_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94ea5fa4afa4317b6e15da5953e3513" is_pre_election: true
I20260923 19:32:28.828348 13571 raft_consensus.cc:2454] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83fe43bb2bdc4254a9cf90d53cf433a2 in term 0.
I20260923 19:32:28.828974 13570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ceb2acfd1534202a9f26f8701d22560" candidate_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94ea5fa4afa4317b6e15da5953e3513"
I20260923 19:32:28.829231 13570 raft_consensus.cc:3037] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:28.832828 13570 raft_consensus.cc:2454] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83fe43bb2bdc4254a9cf90d53cf433a2 in term 1.
I20260923 19:32:28.834450 13780 raft_consensus.cc:686] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 1 LEADER]: Becoming Leader. State: Replica: 83fe43bb2bdc4254a9cf90d53cf433a2, State: Running, Role: LEADER
I20260923 19:32:28.835155 13780 consensus_queue.cc:237] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } }
I20260923 19:32:28.841677 13276 catalog_manager.cc:5495] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83fe43bb2bdc4254a9cf90d53cf433a2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83fe43bb2bdc4254a9cf90d53cf433a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35877 } health_report { overall_health: HEALTHY } } }
W20260923 19:32:28.884179 13753 tablet.cc:2369] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:32:28.888921 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.880486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.0.97.1:35877)
0923 19:32:28.880690 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.880697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.881295 (+ 598us) server_negotiation.cc:407] Connection header received
0923 19:32:28.881405 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.881410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.881539 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.881704 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:32:28.882356 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.883466 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.885638 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.886270 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.886409 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:28.887095 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:32:28.887827 (+ 732us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:32:28.887959 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:28.888533 (+ 574us) server_negotiation.cc:299] Negotiation successful
0923 19:32:28.888632 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260923 19:32:28.894788 13474 tablet.cc:2369] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:32:28.901449 13570 raft_consensus.cc:1261] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [term 1 FOLLOWER]: Refusing update from remote peer 83fe43bb2bdc4254a9cf90d53cf433a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:32:28.902882 13780 consensus_queue.cc:1035] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94ea5fa4afa4317b6e15da5953e3513" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:32:28.905583 13707 raft_consensus.cc:1261] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [term 1 FOLLOWER]: Refusing update from remote peer 83fe43bb2bdc4254a9cf90d53cf433a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:32:28.906770 13780 consensus_queue.cc:1035] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c276af6f04420aab17ed79c029b9af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:32:28.936499 13790 mvcc.cc:205] Tried to move back new op lower bound from 7332626222684295168 to 7332626222433796096. Current Snapshot: MvccSnapshot[applied={T|T < 7332626222684295168}]
I20260923 19:32:28.936515 13788 mvcc.cc:205] Tried to move back new op lower bound from 7332626222684295168 to 7332626222433796096. Current Snapshot: MvccSnapshot[applied={T|T < 7332626222684295168}]
I20260923 19:32:28.961958 13789 mvcc.cc:205] Tried to move back new op lower bound from 7332626222684295168 to 7332626222433796096. Current Snapshot: MvccSnapshot[applied={T|T < 7332626222684295168}]
I20260923 19:32:28.976166 13276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:32:28.976500 13276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:32:28.981390 13276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80fb9d2350b34644907af6f281aa9006: Sending DeleteTablet for 3 replicas of tablet 9ceb2acfd1534202a9f26f8701d22560
I20260923 19:32:28.983384 13687 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ceb2acfd1534202a9f26f8701d22560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:32:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37666
I20260923 19:32:28.984256 13551 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ceb2acfd1534202a9f26f8701d22560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:32:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48770
I20260923 19:32:28.985050 13408 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ceb2acfd1534202a9f26f8701d22560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:32:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49100
I20260923 19:32:28.985636 13802 tablet_replica.cc:330] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: stopping tablet replica
I20260923 19:32:28.986186 13802 raft_consensus.cc:2229] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:32:28.986846 13802 raft_consensus.cc:2258] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:32:28.987711 13801 tablet_replica.cc:330] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: stopping tablet replica
I20260923 19:32:28.988319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 13314
I20260923 19:32:28.988658 13801 raft_consensus.cc:2229] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:32:28.989217 13801 raft_consensus.cc:2258] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:32:28.992921 13802 ts_tablet_manager.cc:1905] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:32:28.993254 13801 ts_tablet_manager.cc:1905] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:32:28.994894 13804 tablet_replica.cc:330] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2: stopping tablet replica
I20260923 19:32:28.997418 13804 raft_consensus.cc:2229] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:32:29.000399 13802 ts_tablet_manager.cc:1918] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:32:29.000581 13802 log.cc:1196] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/9ceb2acfd1534202a9f26f8701d22560
I20260923 19:32:29.000746 13801 ts_tablet_manager.cc:1918] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:32:29.000952 13801 log.cc:1196] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/9ceb2acfd1534202a9f26f8701d22560
I20260923 19:32:29.001132 13802 ts_tablet_manager.cc:1939] T 9ceb2acfd1534202a9f26f8701d22560 P c94ea5fa4afa4317b6e15da5953e3513: Deleting consensus metadata
I20260923 19:32:29.001524 13801 ts_tablet_manager.cc:1939] T 9ceb2acfd1534202a9f26f8701d22560 P 55c276af6f04420aab17ed79c029b9af: Deleting consensus metadata
I20260923 19:32:29.002216 13804 raft_consensus.cc:2258] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:32:29.003688 13264 catalog_manager.cc:4841] TS 55c276af6f04420aab17ed79c029b9af (127.0.97.3:35159): tablet 9ceb2acfd1534202a9f26f8701d22560 (table test-table [id=8f2176a61ddb426bb9bfe2a45f3af6bf]) successfully deleted
I20260923 19:32:29.005275 13262 catalog_manager.cc:4841] TS c94ea5fa4afa4317b6e15da5953e3513 (127.0.97.2:33173): tablet 9ceb2acfd1534202a9f26f8701d22560 (table test-table [id=8f2176a61ddb426bb9bfe2a45f3af6bf]) successfully deleted
I20260923 19:32:29.005662 13804 ts_tablet_manager.cc:1905] T 9ceb2acfd1534202a9f26f8701d22560 P 83fe43bb2bdc4254a9cf90d53cf433a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:32:29.009234 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:28.989618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.0.97.1:35877)
0923 19:32:28.989788 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:28.989794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:28.991788 (+ 1994us) server_negotiation.cc:407] Connection header received
0923 19:32:28.992000 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:28.992006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:28.992098 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:28.992260 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:28.994124 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.995303 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.997379 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:28.998034 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:28.998181 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:29.002017 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:29.002036 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:29.002055 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:29.002135 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:29.006173 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:29.007011 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:29.007020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:29.007024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:29.007200 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:29.007563 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:29.007572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:29.007578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:29.007966 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:29.008310 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:29.008614 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 19:32:29.008870 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 19:32:29.009965 13468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:29.114037 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 13314
I20260923 19:32:29.130918 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 13477
W20260923 19:32:29.131018 13263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35877 (error 108)
I20260923 19:32:29.133119 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:29.132264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35877 (local address 127.0.0.1:39720)
0923 19:32:29.132845 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:29.132948 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":362,"threads_started":1}
W20260923 19:32:29.133440 13263 catalog_manager.cc:4584] TS 83fe43bb2bdc4254a9cf90d53cf433a2 (127.0.97.1:35877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ceb2acfd1534202a9f26f8701d22560: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35877: connect: Connection refused (error 111)
I20260923 19:32:29.146605 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:29.131989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.0.97.2:33173)
0923 19:32:29.132169 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:29.132173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:29.132193 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:32:29.132258 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:29.132262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:29.132328 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:29.132430 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:29.134092 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:29.134864 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:29.136263 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:29.136669 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:29.136765 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:29.139728 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:29.139743 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:29.139759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:29.139801 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:29.143172 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:29.144095 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:29.144105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:29.144108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:29.144388 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:29.144918 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:29.144922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:29.144925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:29.145441 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:29.145681 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:29.146044 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 19:32:29.146327 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 19:32:29.147431 13611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:29.187131 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:29.186691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35877 (local address 127.0.0.1:39722)
0923 19:32:29.186903 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:29.186995 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 19:32:29.188181 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:29.187877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35877 (local address 127.0.0.1:39732)
0923 19:32:29.188013 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:29.188074 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 19:32:29.244951 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 13477
I20260923 19:32:29.248724 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:29.248301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35877 (local address 127.0.0.1:39734)
0923 19:32:29.248491 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:29.248569 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 19:32:29.250020 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:29.249667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35877 (local address 127.0.0.1:39738)
0923 19:32:29.249805 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:29.249887 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 19:32:29.260639 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 13620
I20260923 19:32:29.273854 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:29.261468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.0.97.3:35159)
0923 19:32:29.261678 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:29.261684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:29.261704 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:32:29.261768 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:29.261772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:29.261846 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:29.261927 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:29.263285 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:29.263999 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:29.265283 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:29.265697 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:29.265801 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:29.268399 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:29.268414 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:29.268430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:29.268464 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:29.271116 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:29.271814 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:29.271821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:29.271824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:29.271973 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:29.272329 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:29.272333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:29.272336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:29.272707 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:29.272988 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:29.273292 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 19:32:29.273503 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 19:32:29.274886 13747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:29.331151 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:29.330595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35877 (local address 127.0.0.1:39754)
0923 19:32:29.330849 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:29.330962 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 19:32:29.332605 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:29.332115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35877 (local address 127.0.0.1:39760)
0923 19:32:29.332360 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:29.332457 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 19:32:29.370143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 13620
I20260923 19:32:29.386072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 13244
I20260923 19:32:29.387526 13307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:29.500262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 13244
I20260923 19:32:29.522737 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 19:32:29.858511 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 19:32:29.858752 388 mini_ranger.cc:66] Stopping Ranger...
I20260923 19:32:30.205137 388 mini_ranger.cc:68] Stopped Ranger
2026-09-23 19:32:30.205 UTC [12296] LOG: received smart shutdown request
2026-09-23 19:32:30.209 UTC [12296] LOG: background worker "logical replication launcher" (PID 12305) exited with exit code 1
2026-09-23 19:32:30.210 UTC [12300] LOG: shutting down
2026-09-23 19:32:30.388 UTC [12296] LOG: database system is shut down
2026-09-23T19:32:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54526 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:32:30 dist-test-slave-kxfb krb5kdc[13823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:32:30 dist-test-slave-kxfb krb5kdc[13823](info): set up 2 sockets
Sep 23 19:32:30 dist-test-slave-kxfb krb5kdc[13823](info): commencing operation
krb5kdc: starting...
W20260923 19:32:31.538800 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:32:31 dist-test-slave-kxfb krb5kdc[13823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:32:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:32:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:32:31.789791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37299
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37299,127.0.97.61:38221,127.0.97.60:32881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:32.118271 13836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:32.118623 13836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:32.118758 13836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:32.125988 13836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:32:32.126106 13836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:32.126183 13836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:32.126260 13836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:32:32.126341 13836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:32:32.136274 13836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37299,127.0.97.61:38221,127.0.97.60:32881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:32.137576 13836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:32.138877 13836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:32.151217 13841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:32.151396 13844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:32.151316 13842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:32.152721 13836 server_base.cc:1034] running on GCE node
I20260923 19:32:32.153751 13836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:32.155561 13836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:32.156785 13836 hybrid_clock.cc:648] HybridClock initialized: now 1790191952156739 us; error 45 us; skew 500 ppm
I20260923 19:32:32.157166 13836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:32 dist-test-slave-kxfb krb5kdc[13823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:32.160741 13836 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:32:32.162467 13836 webserver.cc:460] Webserver started at http://127.0.97.62:35337/ using document root <none> and password file <none>
I20260923 19:32:32.163269 13836 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:32.163388 13836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:32.163723 13836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:32:32.167142 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f"
format_stamp: "Formatted at 2026-09-23 19:32:32 on dist-test-slave-kxfb"
server_key: "795be16504355512293505d9c4657e17"
server_key_iv: "51399aa71fbfac0d5426d6971f235078"
server_key_version: "encryptionkey@0"
I20260923 19:32:32.167969 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f"
format_stamp: "Formatted at 2026-09-23 19:32:32 on dist-test-slave-kxfb"
server_key: "795be16504355512293505d9c4657e17"
server_key_iv: "51399aa71fbfac0d5426d6971f235078"
server_key_version: "encryptionkey@0"
I20260923 19:32:32.173772 13836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:32:32.177667 13851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:32.179324 13836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:32:32.179515 13836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f"
format_stamp: "Formatted at 2026-09-23 19:32:32 on dist-test-slave-kxfb"
server_key: "795be16504355512293505d9c4657e17"
server_key_iv: "51399aa71fbfac0d5426d6971f235078"
server_key_version: "encryptionkey@0"
I20260923 19:32:32.179718 13836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:32.212282 13836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:32.217486 13836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:32.217816 13836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:32.238430 13836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37299
I20260923 19:32:32.238444 13902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37299 every 8 connection(s)
I20260923 19:32:32.239982 13836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:32:32.244017 13903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:32.244783 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 13836
I20260923 19:32:32.245002 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:32:32.245680 388 external_mini_cluster.cc:1428] Setting key 5371cb4f2e1f7f38031f2ff3ee4f543d
I20260923 19:32:32.251031 13903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 23 19:32:32 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:32.280037 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.254186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.0.97.62:37299)
0923 19:32:32.259027 (+ 4841us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:32.259049 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:32.259602 (+ 553us) server_negotiation.cc:407] Connection header received
0923 19:32:32.261702 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:32.261712 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:32.262145 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:32.262467 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:32.263859 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.265884 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.267525 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.268466 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.268629 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:32.272418 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:32.272426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:32.272442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:32.272474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:32.275449 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:32.276132 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:32.276139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:32.276141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:32.276288 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:32.276664 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:32.276668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:32.276670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:32.277245 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:32.277440 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:32.278011 (+ 571us) server_negotiation.cc:299] Negotiation successful
0923 19:32:32.278381 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4511,"spinlock_wait_cycles":54656,"thread_start_us":4101,"threads_started":1}
I20260923 19:32:32.280056 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.254186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37299 (local address 127.0.0.1:47208)
0923 19:32:32.259402 (+ 5216us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.259449 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:32.261233 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:32.262515 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:32.262535 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:32.263178 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:32.263587 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:32.263608 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.266113 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.266117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:32.267351 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:32.267366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.268698 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.268703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:32.268851 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:32.269615 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:32.269640 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:32.272175 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:32.275573 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:32.275585 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:32.275601 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:32.276000 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:32.276397 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:32.276402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:32.276406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:32.276560 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:32.277389 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:32.277398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:32.277623 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 19:32:32.277859 (+ 236us) client_negotiation.cc:240] Negotiation successful
0923 19:32:32.278501 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4971,"mutex_wait_us":139,"thread_start_us":482,"threads_started":1}
I20260923 19:32:32.283505 13903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:32:32.284884 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.284293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37562)
0923 19:32:32.284522 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.284696 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260923 19:32:32.285218 13852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
I20260923 19:32:32.287230 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.286728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37578)
0923 19:32:32.286930 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.287034 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 19:32:32.287549 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260923 19:32:32.298012 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38221
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37299,127.0.97.61:38221,127.0.97.60:32881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:32:32.336941 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } attempt: 1
I20260923 19:32:32.338315 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.337807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37590)
0923 19:32:32.338025 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.338120 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 19:32:32.339651 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.339301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37592)
0923 19:32:32.339437 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.339509 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 19:32:32.339861 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
I20260923 19:32:32.408154 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } attempt: 2
I20260923 19:32:32.409708 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.409089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37604)
0923 19:32:32.409415 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.409537 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260923 19:32:32.410907 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.410512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37610)
0923 19:32:32.410670 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.410772 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260923 19:32:32.411113 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
I20260923 19:32:32.502430 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } attempt: 3
I20260923 19:32:32.503839 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.503292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37616)
0923 19:32:32.503538 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.503655 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 19:32:32.505002 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.504580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37620)
0923 19:32:32.504762 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.504851 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 19:32:32.505234 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
W20260923 19:32:32.626847 13910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:32.627147 13910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:32.627265 13910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:32.634467 13910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:32:32.634584 13910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:32.634641 13910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:32.634730 13910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:32:32.634793 13910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:32:32.644639 13910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37299,127.0.97.61:38221,127.0.97.60:32881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:32.645931 13910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:32.647235 13910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 19:32:32.648505 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } attempt: 4
I20260923 19:32:32.649998 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.649328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37636)
0923 19:32:32.649709 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.649822 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260923 19:32:32.651135 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.650768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37640)
0923 19:32:32.650920 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.651009 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 19:32:32.651319 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38221: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38221: connect: Connection refused (error 111)
W20260923 19:32:32.659411 13915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:32.659643 13916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:32.659688 13918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:32.660991 13910 server_base.cc:1034] running on GCE node
I20260923 19:32:32.661893 13910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:32.663687 13910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:32.664944 13910 hybrid_clock.cc:648] HybridClock initialized: now 1790191952664868 us; error 60 us; skew 500 ppm
I20260923 19:32:32.665414 13910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:32 dist-test-slave-kxfb krb5kdc[13823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:32.674610 13910 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260923 19:32:32.676304 13910 webserver.cc:460] Webserver started at http://127.0.97.61:37299/ using document root <none> and password file <none>
I20260923 19:32:32.677106 13910 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:32.677227 13910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:32.677520 13910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:32:32.680598 13910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/data/instance:
uuid: "1b3667bb54904662803742257aeafee7"
format_stamp: "Formatted at 2026-09-23 19:32:32 on dist-test-slave-kxfb"
server_key: "8aaf38b6f003888f7930758e88e7a9b4"
server_key_iv: "6f1f37e862434db39c87d3c5bcef0b4d"
server_key_version: "encryptionkey@0"
I20260923 19:32:32.681349 13910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/wal/instance:
uuid: "1b3667bb54904662803742257aeafee7"
format_stamp: "Formatted at 2026-09-23 19:32:32 on dist-test-slave-kxfb"
server_key: "8aaf38b6f003888f7930758e88e7a9b4"
server_key_iv: "6f1f37e862434db39c87d3c5bcef0b4d"
server_key_version: "encryptionkey@0"
I20260923 19:32:32.686877 13910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 19:32:32.690737 13925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:32.692355 13910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:32:32.692548 13910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/wal
uuid: "1b3667bb54904662803742257aeafee7"
format_stamp: "Formatted at 2026-09-23 19:32:32 on dist-test-slave-kxfb"
server_key: "8aaf38b6f003888f7930758e88e7a9b4"
server_key_iv: "6f1f37e862434db39c87d3c5bcef0b4d"
server_key_version: "encryptionkey@0"
I20260923 19:32:32.692755 13910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:32.709203 13910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:32.713634 13910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:32.713910 13910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:32.732595 13910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38221
I20260923 19:32:32.732674 13976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38221 every 8 connection(s)
I20260923 19:32:32.733947 13910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
I20260923 19:32:32.738433 13977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:32.742331 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 13910
I20260923 19:32:32.742545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-1/wal/instance
I20260923 19:32:32.743122 388 external_mini_cluster.cc:1428] Setting key a085129cda29a2a5531a5fa4a2cd839e
I20260923 19:32:32.747524 13977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:32:32 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260923 19:32:32.768668 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.751007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.0.97.62:37299)
0923 19:32:32.751187 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:32.751192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:32.752738 (+ 1546us) server_negotiation.cc:407] Connection header received
0923 19:32:32.754924 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:32.754931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:32.755016 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:32.755106 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:32.756619 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.757300 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.758811 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.759219 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.759327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:32.762964 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:32.762969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:32.762972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:32.763003 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:32.765629 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:32.766351 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:32.766355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:32.766357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:32.766471 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:32.766865 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:32.766868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:32.766870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:32.767188 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:32.767331 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:32.768134 (+ 803us) server_negotiation.cc:299] Negotiation successful
0923 19:32:32.768393 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260923 19:32:32.769429 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.751626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37299 (local address 127.0.0.1:47212)
0923 19:32:32.752553 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.752599 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:32.754656 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:32.755358 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:32.755367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:32.755893 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:32.756458 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:32.756473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.757442 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.757446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:32.758681 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:32.758692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.759427 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.759434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:32.759644 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:32.760362 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:32.760385 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:32.762755 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:32.765763 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:32.765772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:32.765787 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:32.766213 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:32.766575 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:32.766580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:32.766584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:32.766754 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:32.767312 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:32.767318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:32.767643 (+ 325us) client_negotiation.cc:769] Sending connection context
0923 19:32:32.767910 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 19:32:32.768299 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":403,"threads_started":1}
I20260923 19:32:32.771561 13977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:32:32 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:32:32.787431 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.772372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37654)
0923 19:32:32.772602 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.772622 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:32.772726 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:32.774057 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:32.774066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:32.774572 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:32.774771 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:32.774782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.776280 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.776286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:32.777442 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:32.777455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.778338 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.778345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:32.778605 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:32.779242 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:32.779261 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:32.781569 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:32.784855 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:32.784860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:32.784893 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:32.785260 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:32.785609 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:32.785614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:32.785618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:32.785754 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:32.786382 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:32.786387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:32.786546 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 19:32:32.786666 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 19:32:32.786975 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":3}
I20260923 19:32:32.787482 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.772682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.0.97.61:38221)
0923 19:32:32.773550 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:32.773557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:32.773584 (+ 27us) server_negotiation.cc:407] Connection header received
0923 19:32:32.773648 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:32.773655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:32.773868 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:32.774006 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:32.774953 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.776117 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.777585 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.778162 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.778344 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:32.781778 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:32.781790 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:32.781805 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:32.781832 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:32.784730 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:32.785386 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:32.785395 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:32.785399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:32.785509 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:32.785864 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:32.785869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:32.785872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:32.786245 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:32.786494 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:32.786819 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 19:32:32.787072 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":704,"thread_start_us":423,"threads_started":1}
I20260923 19:32:32.790449 13977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:32:32.791615 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.791138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33176)
0923 19:32:32.791301 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.791430 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 19:32:32.791852 13926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
I20260923 19:32:32.793490 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.793118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33178)
0923 19:32:32.793286 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.793379 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 19:32:32.793697 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32881: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260923 19:32:32.797714 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32881
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37299,127.0.97.61:38221,127.0.97.60:32881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:32:32.843088 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } attempt: 1
I20260923 19:32:32.844430 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.843921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33188)
0923 19:32:32.844133 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.844246 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 19:32:32.845391 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.845065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33198)
0923 19:32:32.845215 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.845282 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 19:32:32.845574 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32881: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
I20260923 19:32:32.913844 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } attempt: 2
I20260923 19:32:32.915333 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.914778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33214)
0923 19:32:32.915021 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.915136 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 19:32:32.916324 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.915967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33218)
0923 19:32:32.916113 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.916186 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 19:32:32.916524 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32881: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
I20260923 19:32:32.950559 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } attempt: 5
Sep 23 19:32:32 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:32:32.965616 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.951574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.0.97.61:38221)
0923 19:32:32.951802 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:32.951809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:32.951839 (+ 30us) server_negotiation.cc:407] Connection header received
0923 19:32:32.951909 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:32.951913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:32.952002 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:32.952087 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:32.952943 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.953699 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.954890 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.955325 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.955441 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:32.959453 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:32.959458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:32.959462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:32.959498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:32.962450 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:32.963418 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:32.963423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:32.963426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:32.963563 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:32.964039 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:32.964043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:32.964045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:32.964431 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:32.964569 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:32.965078 (+ 509us) server_negotiation.cc:299] Negotiation successful
0923 19:32:32.965339 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 19:32:32.965698 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.951392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37662)
0923 19:32:32.951597 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.951619 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:32.951754 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:32.952130 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:32.952135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:32.952590 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:32.952795 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:32.952804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.953849 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.953854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:32.954755 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:32.954765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:32.955466 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:32.955472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:32.955651 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:32.956378 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:32.956396 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:32.959201 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:32.962649 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:32.962656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:32.962660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:32.963250 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:32.963703 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:32.963708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:32.963712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:32.963887 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:32.964567 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:32.964572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:32.964779 (+ 207us) client_negotiation.cc:769] Sending connection context
0923 19:32:32.964953 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 19:32:32.965286 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 19:32:32.967219 13903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:32:32.968502 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.968006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33228)
0923 19:32:32.968204 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.968334 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 19:32:32.969669 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:32.969305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33242)
0923 19:32:32.969474 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:32.969556 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 19:32:32.969898 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32881: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
I20260923 19:32:33.007741 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } attempt: 3
I20260923 19:32:33.009269 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.008563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33256)
0923 19:32:33.008866 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.009016 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 19:32:33.010778 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.010185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33258)
0923 19:32:33.010525 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.010629 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
W20260923 19:32:33.011040 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32881: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
I20260923 19:32:33.021152 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } attempt: 1
I20260923 19:32:33.022434 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.021915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33266)
0923 19:32:33.022115 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.022206 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 19:32:33.023329 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.023022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33270)
0923 19:32:33.023152 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.023219 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 19:32:33.023663 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32881: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
I20260923 19:32:33.091902 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } attempt: 2
I20260923 19:32:33.093524 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.092768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33286)
0923 19:32:33.093093 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.093267 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260923 19:32:33.094801 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.094450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33298)
0923 19:32:33.094591 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.094660 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 19:32:33.095000 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32881: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
W20260923 19:32:33.122677 13984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:33.122982 13984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:33.123112 13984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:33.130241 13984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:32:33.130383 13984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:33.130457 13984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:33.130532 13984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:32:33.130597 13984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:32:33.140151 13984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37299,127.0.97.61:38221,127.0.97.60:32881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:33.141424 13984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:33.142724 13984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 19:32:33.154330 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } attempt: 4
W20260923 19:32:33.155328 13989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:33.155548 13992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:33.155607 13990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:33.156245 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.155067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33304)
0923 19:32:33.155979 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.156078 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":751}
I20260923 19:32:33.156976 13984 server_base.cc:1034] running on GCE node
I20260923 19:32:33.157455 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.157037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33314)
0923 19:32:33.157219 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.157307 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 19:32:33.157662 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32881: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
I20260923 19:32:33.157795 13984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:33.159250 13984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:33.160462 13984 hybrid_clock.cc:648] HybridClock initialized: now 1790191953160401 us; error 48 us; skew 500 ppm
I20260923 19:32:33.160800 13984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:33 dist-test-slave-kxfb krb5kdc[13823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:33.164026 13984 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260923 19:32:33.165552 13984 webserver.cc:460] Webserver started at http://127.0.97.60:34763/ using document root <none> and password file <none>
I20260923 19:32:33.166218 13984 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:33.166353 13984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:33.166625 13984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:32:33.169472 13984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/data/instance:
uuid: "cae1f8984c8342018a352430881a0663"
format_stamp: "Formatted at 2026-09-23 19:32:33 on dist-test-slave-kxfb"
server_key: "5c8eb824f8728d6577768e3f177be830"
server_key_iv: "519b32dcf6d6e3fe7332be16fd35342c"
server_key_version: "encryptionkey@0"
I20260923 19:32:33.170192 13984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/wal/instance:
uuid: "cae1f8984c8342018a352430881a0663"
format_stamp: "Formatted at 2026-09-23 19:32:33 on dist-test-slave-kxfb"
server_key: "5c8eb824f8728d6577768e3f177be830"
server_key_iv: "519b32dcf6d6e3fe7332be16fd35342c"
server_key_version: "encryptionkey@0"
I20260923 19:32:33.175467 13984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 19:32:33.179033 13999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:33.180634 13984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:32:33.180811 13984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/wal
uuid: "cae1f8984c8342018a352430881a0663"
format_stamp: "Formatted at 2026-09-23 19:32:33 on dist-test-slave-kxfb"
server_key: "5c8eb824f8728d6577768e3f177be830"
server_key_iv: "519b32dcf6d6e3fe7332be16fd35342c"
server_key_version: "encryptionkey@0"
I20260923 19:32:33.181021 13984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:33.201239 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } attempt: 3
I20260923 19:32:33.202584 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.202009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33320)
0923 19:32:33.202261 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.202402 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 19:32:33.203711 13984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:33.203929 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.203495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33336)
0923 19:32:33.203701 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.203793 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260923 19:32:33.204227 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32881: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32881: connect: Connection refused (error 111)
I20260923 19:32:33.207993 13984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:33.208262 13984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:33.227406 13984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:32881
I20260923 19:32:33.227485 14050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:32881 every 8 connection(s)
I20260923 19:32:33.228776 13984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
I20260923 19:32:33.233378 14051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:33.234560 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 13984
I20260923 19:32:33.234778 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790191744267121-388-0/minicluster-data/master-2/wal/instance
I20260923 19:32:33.235394 388 external_mini_cluster.cc:1428] Setting key 76a4920ed258a74f5d5ca4153d51c21a
W20260923 19:32:33.242862 13899 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.97.62:37299
I20260923 19:32:33.244308 14051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:32:33 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 19:32:33 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:33.261948 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.237716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.0.97.60:32881)
0923 19:32:33.238805 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.238823 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.238880 (+ 57us) server_negotiation.cc:407] Connection header received
0923 19:32:33.240872 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.240903 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.241498 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.242021 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.243940 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.245996 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.247792 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.248911 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.249053 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.253458 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.253466 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.253478 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.253527 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.257325 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.258150 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.258157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.258164 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.258424 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.258932 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.258937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.258940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.259500 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.259774 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.260265 (+ 491us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.260749 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":455,"threads_started":1}
I20260923 19:32:33.263180 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.247619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.0.97.62:37299)
0923 19:32:33.247848 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.247855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.248738 (+ 883us) server_negotiation.cc:407] Connection header received
0923 19:32:33.248953 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.248957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.249050 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.249165 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.250382 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.251449 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.252855 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.253449 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.253548 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.257646 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.257653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.257658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.257707 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.260205 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.261158 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.261163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.261167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.261292 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.261641 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.261646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.261650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.261891 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.262022 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.262612 (+ 590us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.262822 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 19:32:33.263289 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.247647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37299 (local address 127.0.0.1:47224)
0923 19:32:33.248567 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.248613 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:33.248824 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:33.249256 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:33.249261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:33.249915 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:33.250194 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.250203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.251612 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.251616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.252715 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.252728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.253622 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.253631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.253854 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.254681 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:33.254712 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:33.257434 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:33.260365 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.260378 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.260385 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.261047 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.261395 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.261400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.261402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.261537 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.262022 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:33.262032 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:33.262279 (+ 247us) client_negotiation.cc:769] Sending connection context
0923 19:32:33.262502 (+ 223us) client_negotiation.cc:240] Negotiation successful
0923 19:32:33.262853 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":408,"threads_started":1}
I20260923 19:32:33.264809 14051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:32:33 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 23 19:32:33 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:33.279232 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.265436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38221 (local address 127.0.0.1:37678)
0923 19:32:33.265668 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.265686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:33.265771 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:33.266235 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:33.266238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:33.266677 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:33.266825 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.266833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.268005 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.268011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.269054 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.269066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.270041 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.270047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.270268 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.271033 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:33.271051 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:33.273338 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:33.276638 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.276644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.276648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.277092 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.277454 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.277459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.277463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.277609 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.278217 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:33.278222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:33.278446 (+ 224us) client_negotiation.cc:769] Sending connection context
0923 19:32:33.278620 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 19:32:33.278938 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260923 19:32:33.279388 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.265690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.0.97.61:38221)
0923 19:32:33.265881 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.265887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.265905 (+ 18us) server_negotiation.cc:407] Connection header received
0923 19:32:33.265983 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.265989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.266076 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.266187 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.266973 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.267839 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.269208 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.269868 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.269966 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.273537 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.273542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.273546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.273583 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.276500 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.277202 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.277207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.277209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.277350 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.277708 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.277713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.277715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.278063 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.278225 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.278771 (+ 546us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.279023 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 19:32:33.280507 14051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:32:33.281042 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.265326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.0.97.62:37299)
0923 19:32:33.265614 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.265619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.265637 (+ 18us) server_negotiation.cc:407] Connection header received
0923 19:32:33.265683 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.265688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.265757 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.265855 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.267404 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.268310 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.269809 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.270469 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.270605 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.275073 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.275078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.275082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.275130 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.277716 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.278560 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.278565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.278568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.278722 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.279179 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.279183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.279187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.279531 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.279766 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.280532 (+ 766us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.280751 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
Sep 23 19:32:33 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 19:32:33 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:32:33.295141 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.281767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.0.97.60:32881)
0923 19:32:33.281944 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.281949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.281967 (+ 18us) server_negotiation.cc:407] Connection header received
0923 19:32:33.282010 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.282014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.282076 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.282185 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.283038 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.283798 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.285139 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.285553 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.285697 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.288992 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.288998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.289002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.289033 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.291656 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.292237 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.292243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.292247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.292351 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.292676 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.292681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.292684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.293021 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.293164 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.294387 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.294809 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":186,"server-negotiator.queue_time_us":61}
I20260923 19:32:33.295213 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.281332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33346)
0923 19:32:33.281490 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.281508 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:33.281908 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:33.282230 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:33.282234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:33.282681 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:33.282885 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.282897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.283948 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.283953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.285013 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.285026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.285876 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.285880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.286059 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.286614 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:33.286631 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:33.288820 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:33.291772 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.291778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.291782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.292137 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.292442 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.292447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.292451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.292568 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.293785 (+ 1217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:33.293790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:33.294015 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 19:32:33.294226 (+ 211us) client_negotiation.cc:240] Negotiation successful
0923 19:32:33.294653 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 19:32:33.298792 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.282964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.0.97.61:38221)
0923 19:32:33.283125 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.283130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.283213 (+ 83us) server_negotiation.cc:407] Connection header received
0923 19:32:33.283369 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.283373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.283442 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.283524 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.284939 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.285876 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.287169 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.287621 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.287742 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.292258 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.292264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.292267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.292307 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.295713 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.296600 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.296607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.296610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.296817 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.297247 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.297252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.297254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.297584 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.297852 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.298270 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.298521 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 19:32:33.300904 14051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663: Bootstrap starting.
I20260923 19:32:33.304083 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:32:33.304401 14051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:33.305362 14051 log.cc:824] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:33.308470 14051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663: No bootstrap required, opened a new log
I20260923 19:32:33.320076 14051 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } }
I20260923 19:32:33.320567 14051 raft_consensus.cc:374] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:33.320686 14051 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae1f8984c8342018a352430881a0663, State: Initialized, Role: FOLLOWER
I20260923 19:32:33.321534 14051 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } }
I20260923 19:32:33.322954 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.323449 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:32:33.325359 14051 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:32:33.328511 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.309597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.0.97.62:37299)
0923 19:32:33.309813 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.309820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.310359 (+ 539us) server_negotiation.cc:407] Connection header received
0923 19:32:33.310692 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.310698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.310797 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.310860 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.312731 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.313528 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.315750 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.316490 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.316596 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.320847 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.320854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.320859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.320910 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.324598 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.325538 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.325543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.325546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.325967 (+ 421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.326458 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.326463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.326467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.326874 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.327154 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.327801 (+ 647us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.328068 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 19:32:33.332114 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.310517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.0.97.61:38221)
0923 19:32:33.310699 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.310704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.314320 (+ 3616us) server_negotiation.cc:407] Connection header received
0923 19:32:33.314600 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.314607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.314694 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.314779 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.317358 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.318461 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.320560 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.321235 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.321328 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.324905 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.324912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.324917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.324962 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.328559 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.329467 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.329473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.329476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.329651 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.330146 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.330151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.330155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.330601 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.330851 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.331450 (+ 599us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.331729 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 19:32:33.341130 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.314381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.0.97.60:32881)
0923 19:32:33.314553 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.314559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.322717 (+ 8158us) server_negotiation.cc:407] Connection header received
0923 19:32:33.323048 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.323055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.323151 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.323230 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.325532 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.326683 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.328778 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.329534 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.329641 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.333380 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.333387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.333393 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.333436 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.337292 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.338125 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.338131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.338135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.338381 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.338963 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.338969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.338972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.339448 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.339774 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.340356 (+ 582us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.340652 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260923 19:32:33.342770 14075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:32:33.342911 14075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:32:33.339143 13984 master_runner.cc:186] Error getting master registration for 127.0.97.62:37299: OK, instance_id { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" instance_seqno: 1790191952232605 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 19:32:33.350255 13984 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 19:32:33.381455 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } attempt: 4
Sep 23 19:32:33 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 19:32:33.397130 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.382183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33354)
0923 19:32:33.382493 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.382515 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:33.382660 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:33.383017 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:33.383022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:33.383554 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:33.383740 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.383750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.385093 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.385099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.386245 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.386257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.387167 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.387173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.387321 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.387995 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:33.388014 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:33.390219 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:33.394366 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.394370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.394374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.394811 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.395339 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.395343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.395346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.395512 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.396196 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:33.396201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:33.396438 (+ 237us) client_negotiation.cc:769] Sending connection context
0923 19:32:33.396557 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 19:32:33.396831 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260923 19:32:33.397584 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.382264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.0.97.60:32881)
0923 19:32:33.382492 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.382498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.382595 (+ 97us) server_negotiation.cc:407] Connection header received
0923 19:32:33.382789 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.382795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.382869 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.382963 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.383880 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.384952 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.386400 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.386991 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.387633 (+ 642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.390433 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.390441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.390447 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.390497 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.394138 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.394966 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.394972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.394976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.395159 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.395646 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.395651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.395654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.396036 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.396195 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.396714 (+ 519us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.396982 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260923 19:32:33.402760 13903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f: Bootstrap starting.
I20260923 19:32:33.406576 13903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:33.407824 13903 log.cc:824] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:33.411666 13903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f: No bootstrap required, opened a new log
I20260923 19:32:33.419723 13903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } }
I20260923 19:32:33.420114 13903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:33.420231 13903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6b2e3c5f7c47ebaee68bcf12f77d7f, State: Initialized, Role: FOLLOWER
I20260923 19:32:33.420900 13903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } }
I20260923 19:32:33.422012 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.422561 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:32:33.423264 13903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [sys.catalog]: configured and running, proceeding with master startup.
W20260923 19:32:33.435520 14089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:32:33.435734 14089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:32:33.435140 13836 master_runner.cc:186] Error getting master registration for 127.0.97.61:38221: OK, instance_id { permanent_uuid: "1b3667bb54904662803742257aeafee7" instance_seqno: 1790191952727207 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 19:32:33.449090 13836 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 19:32:33.456890 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } attempt: 5
W20260923 19:32:33.460813 14005 tablet.cc:2369] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 19:32:33 dist-test-slave-kxfb krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260923 19:32:33.471395 13857 tablet.cc:2369] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:32:33.479235 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.457560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32881 (local address 127.0.0.1:33362)
0923 19:32:33.457965 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:33.457988 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:33.458105 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:33.458902 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:33.458909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:33.459543 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:33.459792 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.459804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.461459 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.461466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.463007 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:33.463023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.464053 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.464060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:33.464280 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.465047 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:33.465072 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:33.468254 (+ 3182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:33.472901 (+ 4647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.472908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.472913 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.473537 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.473996 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:33.474001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:33.474005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:33.474196 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:33.477969 (+ 3773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:33.477974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:33.478188 (+ 214us) client_negotiation.cc:769] Sending connection context
0923 19:32:33.478404 (+ 216us) client_negotiation.cc:240] Negotiation successful
0923 19:32:33.478792 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260923 19:32:33.479287 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:33.458379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.0.97.60:32881)
0923 19:32:33.458577 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:33.458583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:33.458600 (+ 17us) server_negotiation.cc:407] Connection header received
0923 19:32:33.458656 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:33.458661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:33.458736 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:33.458840 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:33.459962 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.461173 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.463173 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:33.463881 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:33.464020 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:33.468507 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:33.468514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:33.468520 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:33.468567 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:33.472685 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.473692 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.473699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.473703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.473861 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:33.474335 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:33.474339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:33.474342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:33.474728 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:33.474910 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:33.478529 (+ 3619us) server_negotiation.cc:299] Negotiation successful
0923 19:32:33.478859 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 19:32:33.486564 13977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7: Bootstrap starting.
I20260923 19:32:33.491688 13977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:33.493327 13977 log.cc:824] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:33.501394 13977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7: No bootstrap required, opened a new log
I20260923 19:32:33.513484 13977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } }
I20260923 19:32:33.514005 13977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:33.514134 13977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3667bb54904662803742257aeafee7, State: Initialized, Role: FOLLOWER
I20260923 19:32:33.514966 13977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } }
I20260923 19:32:33.517699 13977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:32:33.520582 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.521138 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:32:33.525364 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:32:33.525760 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } }
I20260923 19:32:33.527988 14064 leader_election.cc:288] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d6b2e3c5f7c47ebaee68bcf12f77d7f (127.0.97.62:37299), 1b3667bb54904662803742257aeafee7 (127.0.97.61:38221)
I20260923 19:32:33.529942 13878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cae1f8984c8342018a352430881a0663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" is_pre_election: true
I20260923 19:32:33.530748 13878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae1f8984c8342018a352430881a0663 in term 0.
I20260923 19:32:33.531637 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cae1f8984c8342018a352430881a0663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3667bb54904662803742257aeafee7" is_pre_election: true
I20260923 19:32:33.532303 13952 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cae1f8984c8342018a352430881a0663 in term 0.
I20260923 19:32:33.533182 14000 leader_election.cc:302] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3667bb54904662803742257aeafee7, cae1f8984c8342018a352430881a0663; no voters:
I20260923 19:32:33.534402 14064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:32:33.534575 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:32:33.534675 14064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:33.541149 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } }
I20260923 19:32:33.544948 13878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cae1f8984c8342018a352430881a0663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f"
I20260923 19:32:33.545279 13878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:33.549441 13878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae1f8984c8342018a352430881a0663 in term 1.
I20260923 19:32:33.550084 14000 leader_election.cc:302] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d6b2e3c5f7c47ebaee68bcf12f77d7f, cae1f8984c8342018a352430881a0663; no voters:
I20260923 19:32:33.551647 14101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:32:33.552440 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cae1f8984c8342018a352430881a0663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3667bb54904662803742257aeafee7"
I20260923 19:32:33.552768 13952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:33.556562 13952 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae1f8984c8342018a352430881a0663 in term 1.
I20260923 19:32:33.557250 14064 leader_election.cc:288] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [CANDIDATE]: Term 1 election: Requested vote from peers 3d6b2e3c5f7c47ebaee68bcf12f77d7f (127.0.97.62:37299), 1b3667bb54904662803742257aeafee7 (127.0.97.61:38221)
I20260923 19:32:33.557714 14101 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [term 1 LEADER]: Becoming Leader. State: Replica: cae1f8984c8342018a352430881a0663, State: Running, Role: LEADER
I20260923 19:32:33.558418 14101 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } }
I20260923 19:32:33.576356 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae1f8984c8342018a352430881a0663. Latest consensus state: current_term: 1 leader_uuid: "cae1f8984c8342018a352430881a0663" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.576829 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:33.587119 14106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:32:33.589879 14106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:32:33.612392 13952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [term 1 FOLLOWER]: Refusing update from remote peer cae1f8984c8342018a352430881a0663: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:32:33.612809 13878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [term 1 FOLLOWER]: Refusing update from remote peer cae1f8984c8342018a352430881a0663: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:32:33.613732 14064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:32:33.614475 14064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:32:33.637373 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae1f8984c8342018a352430881a0663. Latest consensus state: current_term: 1 leader_uuid: "cae1f8984c8342018a352430881a0663" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.637745 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [sys.catalog]: This master's current role is: FOLLOWER
W20260923 19:32:33.641387 14107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:32:33.641590 14107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:32:33.648288 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae1f8984c8342018a352430881a0663. Latest consensus state: current_term: 1 leader_uuid: "cae1f8984c8342018a352430881a0663" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.648626 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:32:33.652158 14111 mvcc.cc:205] Tried to move back new op lower bound from 7332626241976537088 to 7332626241782378496. Current Snapshot: MvccSnapshot[applied={T|T < 7332626241976537088}]
I20260923 19:32:33.668025 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cae1f8984c8342018a352430881a0663" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.668408 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:32:33.669148 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cae1f8984c8342018a352430881a0663" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.669451 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:32:33.684176 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cae1f8984c8342018a352430881a0663" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.684597 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:33.685002 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cae1f8984c8342018a352430881a0663" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6b2e3c5f7c47ebaee68bcf12f77d7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37299 } } peers { permanent_uuid: "1b3667bb54904662803742257aeafee7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38221 } } peers { permanent_uuid: "cae1f8984c8342018a352430881a0663" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32881 } } }
I20260923 19:32:33.685237 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:33.688045 14106 catalog_manager.cc:1347] Generated new cluster ID: 7957292604ad4b42897bf41cd2328eba
I20260923 19:32:33.688158 14106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:32:33.715237 14106 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:32:33.717443 14106 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:32:33.742803 14106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663: Generated new TSK 0
I20260923 19:32:33.744127 14106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:32:34.438162 14089 catalog_manager.cc:1259] Loaded cluster ID: 7957292604ad4b42897bf41cd2328eba
I20260923 19:32:34.438338 14089 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f: loading cluster ID for follower catalog manager: success
I20260923 19:32:34.441092 14089 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f: acquiring CA information for follower catalog manager: success
I20260923 19:32:34.443133 14089 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d6b2e3c5f7c47ebaee68bcf12f77d7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 19:32:34.644048 14107 catalog_manager.cc:1259] Loaded cluster ID: 7957292604ad4b42897bf41cd2328eba
I20260923 19:32:34.644184 14107 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7: loading cluster ID for follower catalog manager: success
I20260923 19:32:34.647111 14107 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7: acquiring CA information for follower catalog manager: success
I20260923 19:32:34.649225 14107 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b3667bb54904662803742257aeafee7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 19:32:34.893874 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 13836
I20260923 19:32:34.895490 13898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:35.019608 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 13836
I20260923 19:32:35.040911 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 13910
I20260923 19:32:35.042418 13972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:35.159472 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 13910
I20260923 19:32:35.163003 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:35.162135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37299 (local address 127.0.0.1:47250)
0923 19:32:35.162668 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:35.162791 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":306,"threads_started":1}
W20260923 19:32:35.163338 14000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37299: connect: Connection refused (error 111)
I20260923 19:32:35.167451 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:35.167073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37299 (local address 127.0.0.1:47258)
0923 19:32:35.167223 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:35.167312 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 19:32:35.167661 14000 consensus_peers.cc:487] T 00000000000000000000000000000000 P cae1f8984c8342018a352430881a0663 -> Peer 3d6b2e3c5f7c47ebaee68bcf12f77d7f (127.0.97.62:37299): Couldn't send request to peer 3d6b2e3c5f7c47ebaee68bcf12f77d7f. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 19:32:35.181417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 13984
I20260923 19:32:35.182677 14046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:35.302819 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 13984
2026-09-23T19:32:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4795 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:32:35 dist-test-slave-kxfb krb5kdc[14127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:32:35 dist-test-slave-kxfb krb5kdc[14127](info): set up 2 sockets
Sep 23 19:32:35 dist-test-slave-kxfb krb5kdc[14127](info): commencing operation
krb5kdc: starting...
W20260923 19:32:36.334771 388 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:32:36 dist-test-slave-kxfb krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:32:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:32:36.590072 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36325
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:36.918213 14140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:36.918569 14140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:36.918707 14140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:36.925954 14140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:32:36.926085 14140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:36.926179 14140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:36.926254 14140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:32:36.926331 14140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:32:36.936036 14140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:36.937322 14140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:36.938632 14140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:36.950984 14146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:36.951422 14145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:36.951699 14148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:36.952761 14140 server_base.cc:1034] running on GCE node
I20260923 19:32:36.953589 14140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:36.955098 14140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:36.956312 14140 hybrid_clock.cc:648] HybridClock initialized: now 1790191956956251 us; error 54 us; skew 500 ppm
I20260923 19:32:36.956656 14140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:36 dist-test-slave-kxfb krb5kdc[14127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:36.959967 14140 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:32:36.961572 14140 webserver.cc:460] Webserver started at http://127.0.97.62:33063/ using document root <none> and password file <none>
I20260923 19:32:36.962276 14140 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:36.962446 14140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:36.962761 14140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:32:36.965811 14140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "ab6563604f634d468b75a038359259d2"
format_stamp: "Formatted at 2026-09-23 19:32:36 on dist-test-slave-kxfb"
server_key: "bee6731e27c7c42d33b64e35a0330e77"
server_key_iv: "72a6300a3ba6f2a09d4b36d900a61629"
server_key_version: "encryptionkey@0"
I20260923 19:32:36.966632 14140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "ab6563604f634d468b75a038359259d2"
format_stamp: "Formatted at 2026-09-23 19:32:36 on dist-test-slave-kxfb"
server_key: "bee6731e27c7c42d33b64e35a0330e77"
server_key_iv: "72a6300a3ba6f2a09d4b36d900a61629"
server_key_version: "encryptionkey@0"
I20260923 19:32:36.972442 14140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260923 19:32:36.976191 14155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:36.977839 14140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 19:32:36.978037 14140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "ab6563604f634d468b75a038359259d2"
format_stamp: "Formatted at 2026-09-23 19:32:36 on dist-test-slave-kxfb"
server_key: "bee6731e27c7c42d33b64e35a0330e77"
server_key_iv: "72a6300a3ba6f2a09d4b36d900a61629"
server_key_version: "encryptionkey@0"
I20260923 19:32:36.978226 14140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:37.012667 14140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:37.016958 14140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:37.017243 14140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:37.036859 14140 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36325
I20260923 19:32:37.036959 14206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36325 every 8 connection(s)
I20260923 19:32:37.038406 14140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:32:37.043133 14207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:37.045493 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 14140
I20260923 19:32:37.045708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:32:37.046468 388 external_mini_cluster.cc:1428] Setting key 94cc59340dedee07199c641f8a19245d
I20260923 19:32:37.058221 14207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2: Bootstrap starting.
I20260923 19:32:37.062341 14207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2: Neither blocks nor log segments found. Creating new log.
Sep 23 19:32:37 dist-test-slave-kxfb krb5kdc[14127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:37.063562 14207 log.cc:824] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:37.068068 14207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2: No bootstrap required, opened a new log
I20260923 19:32:37.073936 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:37.048613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.0.97.62:36325)
0923 19:32:37.049641 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:37.049662 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:37.049746 (+ 84us) server_negotiation.cc:407] Connection header received
0923 19:32:37.052341 (+ 2595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:37.052377 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:37.052987 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:37.053497 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:37.055053 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:37.057159 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:37.058649 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:37.059705 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:37.059916 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:37.064261 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:37.064275 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:37.064305 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:37.064364 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:37.068466 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:37.069300 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:37.069309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:37.069319 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:37.069575 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:37.069945 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:37.069950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:37.069953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:37.070919 (+ 966us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:37.071266 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:37.071687 (+ 421us) server_negotiation.cc:299] Negotiation successful
0923 19:32:37.072262 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":439,"threads_started":1}
I20260923 19:32:37.079054 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:32:37.079063 14207 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab6563604f634d468b75a038359259d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36325 } }
I20260923 19:32:37.079506 14207 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:37.079612 14207 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6563604f634d468b75a038359259d2, State: Initialized, Role: FOLLOWER
I20260923 19:32:37.080226 14207 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab6563604f634d468b75a038359259d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36325 } }
I20260923 19:32:37.080475 14207 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:32:37.080615 14207 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:32:37.080781 14207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:37.083806 14207 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab6563604f634d468b75a038359259d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36325 } }
I20260923 19:32:37.084429 14207 leader_election.cc:302] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab6563604f634d468b75a038359259d2; no voters:
I20260923 19:32:37.085085 14207 leader_election.cc:288] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:32:37.085281 14212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:32:37.086046 14212 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [term 1 LEADER]: Becoming Leader. State: Replica: ab6563604f634d468b75a038359259d2, State: Running, Role: LEADER
I20260923 19:32:37.086730 14212 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab6563604f634d468b75a038359259d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36325 } }
I20260923 19:32:37.087545 14207 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:32:37.089780 14214 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab6563604f634d468b75a038359259d2. Latest consensus state: current_term: 1 leader_uuid: "ab6563604f634d468b75a038359259d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab6563604f634d468b75a038359259d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36325 } } }
I20260923 19:32:37.090205 14214 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:37.090406 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab6563604f634d468b75a038359259d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab6563604f634d468b75a038359259d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36325 } } }
I20260923 19:32:37.090633 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:37.091507 14219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:32:37.099713 14219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:32:37.110797 14219 catalog_manager.cc:1347] Generated new cluster ID: 96bb94e4b2c44ff2bc3c28b0de73d907
I20260923 19:32:37.110961 14219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:32:37.139106 14219 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:32:37.140704 14219 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:32:37.147020 14219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab6563604f634d468b75a038359259d2: Generated new TSK 0
I20260923 19:32:37.148172 14219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:32:37.193723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 14140
I20260923 19:32:37.195246 14202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:37.293511 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 14140
2026-09-23T19:32:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1987 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:32:37 dist-test-slave-kxfb krb5kdc[14235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:32:37 dist-test-slave-kxfb krb5kdc[14235](info): set up 2 sockets
Sep 23 19:32:37 dist-test-slave-kxfb krb5kdc[14235](info): commencing operation
krb5kdc: starting...
W20260923 19:32:38.314620 388 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:32:38 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:32:38.566207 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:38.891232 14248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:38.891556 14248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:38.891691 14248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:38.899250 14248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:32:38.899381 14248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:38.899459 14248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:38.899537 14248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:32:38.899600 14248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:32:38.910585 14248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:38.911945 14248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:38.913228 14248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:38.925618 14254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:38.925717 14256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:38.925664 14253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:38.926669 14248 server_base.cc:1034] running on GCE node
I20260923 19:32:38.927523 14248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:38.929060 14248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:38.930348 14248 hybrid_clock.cc:648] HybridClock initialized: now 1790191958930246 us; error 49 us; skew 500 ppm
I20260923 19:32:38.930724 14248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:38 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:38.940219 14248 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:32:38.941964 14248 webserver.cc:460] Webserver started at http://127.0.97.62:39987/ using document root <none> and password file <none>
I20260923 19:32:38.942737 14248 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:38.942857 14248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:38.943156 14248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:32:38.946166 14248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "8134fe0d94a64de6a110832539a4f405"
format_stamp: "Formatted at 2026-09-23 19:32:38 on dist-test-slave-kxfb"
server_key: "b97fba90af9299ed56090554b9b6476c"
server_key_iv: "799a1693aae4c8baffb3c9493763a605"
server_key_version: "encryptionkey@0"
I20260923 19:32:38.946934 14248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "8134fe0d94a64de6a110832539a4f405"
format_stamp: "Formatted at 2026-09-23 19:32:38 on dist-test-slave-kxfb"
server_key: "b97fba90af9299ed56090554b9b6476c"
server_key_iv: "799a1693aae4c8baffb3c9493763a605"
server_key_version: "encryptionkey@0"
I20260923 19:32:38.952308 14248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:32:38.956032 14263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:38.957566 14248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 19:32:38.957741 14248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "8134fe0d94a64de6a110832539a4f405"
format_stamp: "Formatted at 2026-09-23 19:32:38 on dist-test-slave-kxfb"
server_key: "b97fba90af9299ed56090554b9b6476c"
server_key_iv: "799a1693aae4c8baffb3c9493763a605"
server_key_version: "encryptionkey@0"
I20260923 19:32:38.957916 14248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:38.985235 14248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:38.989374 14248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:38.989646 14248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:39.008941 14248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43013
I20260923 19:32:39.009017 14314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43013 every 8 connection(s)
I20260923 19:32:39.010392 14248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:32:39.015106 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:39.020946 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 14248
I20260923 19:32:39.021209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:32:39.022094 388 external_mini_cluster.cc:1428] Setting key 935590ba85b8b3c77c232f7e939c6d46
I20260923 19:32:39.030043 14315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Bootstrap starting.
I20260923 19:32:39.034559 14315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:39.035952 14315 log.cc:824] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Log is configured to *not* fsync() on all Append() calls
Sep 23 19:32:39 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:39.039810 14315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: No bootstrap required, opened a new log
I20260923 19:32:39.048725 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:39.024808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.0.97.62:43013)
0923 19:32:39.025783 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:39.025796 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:39.025845 (+ 49us) server_negotiation.cc:407] Connection header received
0923 19:32:39.028038 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:39.028063 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:39.028487 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:39.028847 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:39.030946 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:39.032342 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:39.034376 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:39.035145 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:39.035292 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:39.040309 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:39.040327 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:39.040346 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:39.040397 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:39.044144 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:39.044977 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:39.044984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:39.044992 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:39.045194 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:39.045583 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:39.045587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:39.045589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:39.046242 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:39.046628 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:39.047103 (+ 475us) server_negotiation.cc:299] Negotiation successful
0923 19:32:39.047587 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":444,"threads_started":1}
I20260923 19:32:39.049232 14315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:39.049816 14315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:39.049933 14315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8134fe0d94a64de6a110832539a4f405, State: Initialized, Role: FOLLOWER
I20260923 19:32:39.050700 14315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:39.050971 14315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:32:39.051106 14315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:32:39.051285 14315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:39.053975 14315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:39.054663 14315 leader_election.cc:302] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8134fe0d94a64de6a110832539a4f405; no voters:
I20260923 19:32:39.055363 14315 leader_election.cc:288] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:32:39.055526 14320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:32:39.056354 14320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 1 LEADER]: Becoming Leader. State: Replica: 8134fe0d94a64de6a110832539a4f405, State: Running, Role: LEADER
I20260923 19:32:39.056967 14320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:39.057854 14315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:32:39.060645 14322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8134fe0d94a64de6a110832539a4f405. Latest consensus state: current_term: 1 leader_uuid: "8134fe0d94a64de6a110832539a4f405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260923 19:32:39.060707 14321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8134fe0d94a64de6a110832539a4f405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260923 19:32:39.061328 14321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:39.062049 14322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:39.062428 14329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:32:39.067819 14329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:32:39.079002 14329 catalog_manager.cc:1347] Generated new cluster ID: 76b60dec10ff4e8488346fbbfe9f6e2e
I20260923 19:32:39.079231 14329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:32:39.095651 14329 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:32:39.097354 14329 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:32:39.107777 14329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Generated new TSK 0
I20260923 19:32:39.108919 14329 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:32:39.178076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:39.572947 14341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:39.573282 14341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:39.573421 14341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:39.580854 14341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:39.581004 14341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:39.581168 14341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:32:39.591557 14341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:39.592947 14341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:39.594369 14341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:39.608916 14347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:39.609094 14346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:39.609073 14349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:39.609661 14341 server_base.cc:1034] running on GCE node
I20260923 19:32:39.610425 14341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:39.611500 14341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:39.612732 14341 hybrid_clock.cc:648] HybridClock initialized: now 1790191959612672 us; error 66 us; skew 500 ppm
I20260923 19:32:39.613085 14341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:39 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:39.616436 14341 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:32:39.617998 14341 webserver.cc:460] Webserver started at http://127.0.97.1:37931/ using document root <none> and password file <none>
I20260923 19:32:39.618769 14341 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:39.618891 14341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:39.619201 14341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:32:39.622193 14341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "8f1c8e4712f641288dde8ce6427a8a7e"
format_stamp: "Formatted at 2026-09-23 19:32:39 on dist-test-slave-kxfb"
server_key: "d75a6a2ccec94d36cd0a3e50e37ccd32"
server_key_iv: "29bc918032cfdfae9bef4021a6450beb"
server_key_version: "encryptionkey@0"
I20260923 19:32:39.623006 14341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8f1c8e4712f641288dde8ce6427a8a7e"
format_stamp: "Formatted at 2026-09-23 19:32:39 on dist-test-slave-kxfb"
server_key: "d75a6a2ccec94d36cd0a3e50e37ccd32"
server_key_iv: "29bc918032cfdfae9bef4021a6450beb"
server_key_version: "encryptionkey@0"
I20260923 19:32:39.628389 14341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:32:39.632272 14356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:39.634160 14341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 19:32:39.634419 14341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "8f1c8e4712f641288dde8ce6427a8a7e"
format_stamp: "Formatted at 2026-09-23 19:32:39 on dist-test-slave-kxfb"
server_key: "d75a6a2ccec94d36cd0a3e50e37ccd32"
server_key_iv: "29bc918032cfdfae9bef4021a6450beb"
server_key_version: "encryptionkey@0"
I20260923 19:32:39.634660 14341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:39.646803 14341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:39.651038 14341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:39.651335 14341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:39.652356 14341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:39.653821 14341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:39.653898 14341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:39.654008 14341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:39.654060 14341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:39.688699 14341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42795
I20260923 19:32:39.688731 14468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42795 every 8 connection(s)
I20260923 19:32:39.689996 14341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:32:39.698709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 14341
I20260923 19:32:39.698921 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:32:39.699744 388 external_mini_cluster.cc:1428] Setting key fd704006e4e3671ce720147ac956e718
Sep 23 19:32:39 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:39.714567 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:39.693086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51923 (local address 127.0.97.62:43013)
0923 19:32:39.693620 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:39.693625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:39.694564 (+ 939us) server_negotiation.cc:407] Connection header received
0923 19:32:39.696689 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:39.696695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:39.696792 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:39.696902 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:39.698976 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:39.700166 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:39.702231 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:39.702849 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:39.702979 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:39.707411 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:39.707418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:39.707422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:39.707464 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:39.710729 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:39.711775 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:39.711782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:39.711786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:39.711947 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:39.712420 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:39.712423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:39.712427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:39.712822 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:39.712986 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:39.713868 (+ 882us) server_negotiation.cc:299] Negotiation successful
0923 19:32:39.714151 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
I20260923 19:32:39.715437 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:39.693496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:51923)
0923 19:32:39.694371 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:39.694427 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:39.696368 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:39.697199 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:39.697210 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:39.697943 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:39.698702 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:39.698723 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:39.700379 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:39.700385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:39.702054 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:39.702069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:39.703096 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:39.703102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:39.703334 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:39.704086 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:39.704112 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:39.707058 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:39.710983 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:39.710993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:39.711010 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:39.711617 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:39.712098 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:39.712101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:39.712103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:39.712309 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:39.712993 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:39.713000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:39.713445 (+ 445us) client_negotiation.cc:769] Sending connection context
0923 19:32:39.713706 (+ 261us) client_negotiation.cc:240] Negotiation successful
0923 19:32:39.714213 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":315,"threads_started":1}
I20260923 19:32:39.717204 14469 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43013
I20260923 19:32:39.717664 14469 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:39.718734 14469 heartbeater.cc:511] Master 127.0.97.62:43013 requested a full tablet report, sending...
I20260923 19:32:39.721666 14280 ts_manager.cc:194] Registered new tserver with Master: 8f1c8e4712f641288dde8ce6427a8a7e (127.0.97.1:42795)
I20260923 19:32:39.724028 14280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51923
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:32:39.757230 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:40.127576 14475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:40.127948 14475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:40.128093 14475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:40.136291 14475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:40.136451 14475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:40.136746 14475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:32:40.147953 14475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:40.149647 14475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:40.151237 14475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:40.166159 14483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:40.166201 14481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:40.166122 14480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:40.167511 14475 server_base.cc:1034] running on GCE node
I20260923 19:32:40.168286 14475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:40.169404 14475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:40.170653 14475 hybrid_clock.cc:648] HybridClock initialized: now 1790191960170570 us; error 72 us; skew 500 ppm
I20260923 19:32:40.171034 14475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:40 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:40.174553 14475 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:32:40.176306 14475 webserver.cc:460] Webserver started at http://127.0.97.2:37731/ using document root <none> and password file <none>
I20260923 19:32:40.177098 14475 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:40.177203 14475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:40.177495 14475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:32:40.180927 14475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "c4b83ca6fbd643cb931810f78512ad66"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "caa2afe31adcd2e534e3cb4da8b104e7"
server_key_iv: "a6b86d66706b897343d589d388a2444c"
server_key_version: "encryptionkey@0"
I20260923 19:32:40.181777 14475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c4b83ca6fbd643cb931810f78512ad66"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "caa2afe31adcd2e534e3cb4da8b104e7"
server_key_iv: "a6b86d66706b897343d589d388a2444c"
server_key_version: "encryptionkey@0"
I20260923 19:32:40.187906 14475 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 19:32:40.192530 14490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:40.194247 14475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:32:40.194483 14475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "c4b83ca6fbd643cb931810f78512ad66"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "caa2afe31adcd2e534e3cb4da8b104e7"
server_key_iv: "a6b86d66706b897343d589d388a2444c"
server_key_version: "encryptionkey@0"
I20260923 19:32:40.194707 14475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:40.225407 14475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:40.230515 14475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:40.230839 14475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:40.232023 14475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:40.233585 14475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:40.233662 14475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:40.233790 14475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:40.233840 14475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:40.269032 14475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44133
I20260923 19:32:40.269100 14602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44133 every 8 connection(s)
I20260923 19:32:40.270571 14475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:32:40.272883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 14475
I20260923 19:32:40.273066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:32:40.273643 388 external_mini_cluster.cc:1428] Setting key e08885c930f6f8cf1ec9e167829b2ecd
Sep 23 19:32:40 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:40.295349 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.273609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42625 (local address 127.0.97.62:43013)
0923 19:32:40.274747 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:40.274753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:40.275392 (+ 639us) server_negotiation.cc:407] Connection header received
0923 19:32:40.277948 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:40.277953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:40.278035 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:40.278144 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:40.280358 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.281228 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.283498 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.283937 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.284054 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.288202 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:40.288209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:40.288213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:40.288261 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:40.291676 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:40.292585 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:40.292590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:40.292593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:40.292770 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:40.293256 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:40.293260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:40.293263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:40.293642 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:40.293790 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:40.294652 (+ 862us) server_negotiation.cc:299] Negotiation successful
0923 19:32:40.294939 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":831,"threads_started":1}
I20260923 19:32:40.296160 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.274234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:42625)
0923 19:32:40.275195 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:40.275243 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:40.277589 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:40.278488 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:40.278500 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:40.279269 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:40.280123 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.280150 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.281405 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.281410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.283330 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.283346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.284202 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.284208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.284490 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.285254 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:40.285279 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:40.287896 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:40.291893 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:40.291901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:40.291916 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:40.292441 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:40.292892 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:40.292896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:40.292898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:40.293144 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:40.293782 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:40.293788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:40.294197 (+ 409us) client_negotiation.cc:769] Sending connection context
0923 19:32:40.294515 (+ 318us) client_negotiation.cc:240] Negotiation successful
0923 19:32:40.294959 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":379,"threads_started":1}
I20260923 19:32:40.297808 14603 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43013
I20260923 19:32:40.298249 14603 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:40.299245 14603 heartbeater.cc:511] Master 127.0.97.62:43013 requested a full tablet report, sending...
I20260923 19:32:40.300922 14280 ts_manager.cc:194] Registered new tserver with Master: c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 19:32:40.301818 14280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:32:40.335192 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:40.684283 14609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:40.684576 14609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:40.684697 14609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:40.692353 14609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:40.692481 14609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:40.692634 14609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:32:40.702865 14609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:40.704224 14609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:40.705523 14609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:40.719933 14615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:40.719974 14614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:40.719957 14617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:40.723986 14609 server_base.cc:1034] running on GCE node
I20260923 19:32:40.724746 14609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:40.725839 14609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:40.727099 14609 hybrid_clock.cc:648] HybridClock initialized: now 1790191960727028 us; error 67 us; skew 500 ppm
I20260923 19:32:40.727465 14609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:32:40.728474 14469 heartbeater.cc:503] Master 127.0.97.62:43013 was elected leader, sending a full tablet report...
Sep 23 19:32:40 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:40.731528 14609 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:32:40.733570 14609 webserver.cc:460] Webserver started at http://127.0.97.3:42311/ using document root <none> and password file <none>
I20260923 19:32:40.734581 14609 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:40.734737 14609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:40.735122 14609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:32:40.739081 14609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "17532166823145ab87ea861bbc005d3f"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "cfa0b0bb54a1e445316cdc6e1f27de13"
server_key_iv: "bc93b09e212e680de78f0360326869e6"
server_key_version: "encryptionkey@0"
I20260923 19:32:40.740151 14609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "17532166823145ab87ea861bbc005d3f"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "cfa0b0bb54a1e445316cdc6e1f27de13"
server_key_iv: "bc93b09e212e680de78f0360326869e6"
server_key_version: "encryptionkey@0"
I20260923 19:32:40.747017 14609 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 19:32:40.751448 14624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:40.753237 14609 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20260923 19:32:40.753496 14609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "17532166823145ab87ea861bbc005d3f"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "cfa0b0bb54a1e445316cdc6e1f27de13"
server_key_iv: "bc93b09e212e680de78f0360326869e6"
server_key_version: "encryptionkey@0"
I20260923 19:32:40.753718 14609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:40.799299 14609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:40.804545 14609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:40.804883 14609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:40.806082 14609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:40.807760 14609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:40.807845 14609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:40.807976 14609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:40.808043 14609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:40.844594 14609 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42093
I20260923 19:32:40.844614 14736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42093 every 8 connection(s)
I20260923 19:32:40.846228 14609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:32:40.851909 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 14609
I20260923 19:32:40.852126 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:32:40.852588 388 external_mini_cluster.cc:1428] Setting key e58a9a917e8bce6f1b46f644350df439
Sep 23 19:32:40 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:40.867906 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.849266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55205 (local address 127.0.97.62:43013)
0923 19:32:40.849831 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:40.849838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:40.850815 (+ 977us) server_negotiation.cc:407] Connection header received
0923 19:32:40.853006 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:40.853012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:40.853122 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:40.853231 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:40.855034 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.855906 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.857583 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.858021 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.858135 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.861921 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:40.861926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:40.861930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:40.861965 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:40.864646 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:40.865371 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:40.865375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:40.865378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:40.865496 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:40.865919 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:40.865922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:40.865925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:40.866235 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:40.866441 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:40.867214 (+ 773us) server_negotiation.cc:299] Negotiation successful
0923 19:32:40.867481 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":265,"threads_started":1}
I20260923 19:32:40.869047 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.849776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:55205)
0923 19:32:40.850628 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:40.850677 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:40.852677 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:40.853528 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:40.853538 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:40.854128 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:40.854858 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.854877 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.856079 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.856083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.857439 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.857452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.858281 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.858317 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.858559 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.859287 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:40.859312 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:40.861697 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:40.864780 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:40.864790 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:40.864810 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:40.865240 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:40.865594 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:40.865599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:40.865603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:40.865798 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:40.866404 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:40.866413 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:40.866801 (+ 388us) client_negotiation.cc:769] Sending connection context
0923 19:32:40.867066 (+ 265us) client_negotiation.cc:240] Negotiation successful
0923 19:32:40.867729 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":328,"threads_started":1}
I20260923 19:32:40.870561 14737 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43013
I20260923 19:32:40.870946 14737 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:40.871758 14737 heartbeater.cc:511] Master 127.0.97.62:43013 requested a full tablet report, sending...
I20260923 19:32:40.873188 14280 ts_manager.cc:194] Registered new tserver with Master: 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093)
I20260923 19:32:40.874007 14280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55205
I20260923 19:32:40.879630 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:32:40.897702 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.884306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.0.97.62:43013)
0923 19:32:40.884486 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:40.884491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:40.884915 (+ 424us) server_negotiation.cc:407] Connection header received
0923 19:32:40.885114 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:40.885118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:40.885188 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:40.885285 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:40.886801 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.887561 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.888936 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.889397 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.889514 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.892252 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:40.892257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:40.892261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:40.892300 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:40.895044 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:40.895840 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:40.895844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:40.895846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:40.895998 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:40.896355 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:40.896358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:40.896360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:40.896717 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:40.896915 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:40.897241 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 19:32:40.897440 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 19:32:40.902458 14280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:32:40.909595 14280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:32:40.949954 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.925591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44133 (local address 127.0.0.1:34216)
0923 19:32:40.941265 (+ 15674us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:40.941290 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:40.941418 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:40.942453 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:40.942463 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:40.942494 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:40.942965 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.942980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.944976 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.944982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.946553 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.946570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.948659 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.948666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.948997 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.949103 (+ 106us) client_negotiation.cc:769] Sending connection context
0923 19:32:40.949236 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 19:32:40.949483 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15550,"mutex_wait_us":137,"thread_start_us":13483,"threads_started":1}
I20260923 19:32:40.950894 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.925939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.0.97.2:44133)
0923 19:32:40.940809 (+ 14870us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:40.940826 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:40.941537 (+ 711us) server_negotiation.cc:407] Connection header received
0923 19:32:40.941671 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:40.941680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:40.942057 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:40.942268 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:40.943324 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.944831 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.946739 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.948239 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.948367 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.948629 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:40.950323 (+ 1694us) server_negotiation.cc:299] Negotiation successful
0923 19:32:40.950549 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14694,"thread_start_us":14580,"threads_started":1}
I20260923 19:32:40.951648 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.924063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43774)
0923 19:32:40.925067 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:40.925124 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:40.925352 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:40.944309 (+ 18957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:40.944315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:40.944345 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:40.944814 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.944828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.946787 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.946792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.949087 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.949103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.950742 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.950748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.950913 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.950949 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 19:32:40.951059 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 19:32:40.951209 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":38,"thread_start_us":567,"threads_started":1}
I20260923 19:32:40.952323 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.924369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42093 (local address 127.0.0.1:52442)
0923 19:32:40.939812 (+ 15443us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:40.939850 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:40.940079 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:40.943884 (+ 3805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:40.943891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:40.943936 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:40.944389 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.944404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.946613 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.946618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.949094 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:40.949103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.951456 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.951463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:40.951660 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.951690 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 19:32:40.951945 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 19:32:40.952073 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15249,"mutex_wait_us":140,"thread_start_us":16650,"threads_started":1}
I20260923 19:32:40.953086 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.925118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.0.97.3:42093)
0923 19:32:40.943195 (+ 18077us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:40.943205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:40.943240 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:32:40.943351 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:40.943359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:40.943624 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:40.943840 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:40.944642 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.946454 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.949260 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.951235 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.951385 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.951620 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:40.952590 (+ 970us) server_negotiation.cc:299] Negotiation successful
0923 19:32:40.952792 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17849,"thread_start_us":17654,"threads_started":1}
I20260923 19:32:40.954622 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:40.924725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.0.97.1:42795)
0923 19:32:40.943759 (+ 19034us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:40.943768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:40.943797 (+ 29us) server_negotiation.cc:407] Connection header received
0923 19:32:40.943904 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:40.943911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:40.944111 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:40.944290 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:40.945036 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.946593 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.949315 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:40.950522 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:40.950662 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:40.950911 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:40.951882 (+ 971us) server_negotiation.cc:299] Negotiation successful
0923 19:32:40.952099 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18840,"thread_start_us":18655,"threads_started":1}
I20260923 19:32:40.955646 14538 tablet_service.cc:1467] Processing CreateTablet for tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d (DEFAULT_TABLE table=test-table [id=7b90b728ba984d4fbb9039787da67ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:32:40.956833 14672 tablet_service.cc:1467] Processing CreateTablet for tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d (DEFAULT_TABLE table=test-table [id=7b90b728ba984d4fbb9039787da67ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:32:40.957479 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:40.958269 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:40.958249 14404 tablet_service.cc:1467] Processing CreateTablet for tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d (DEFAULT_TABLE table=test-table [id=7b90b728ba984d4fbb9039787da67ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:32:40.959646 14404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:40.971377 14756 tablet_bootstrap.cc:492] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: Bootstrap starting.
I20260923 19:32:40.974145 14758 tablet_bootstrap.cc:492] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: Bootstrap starting.
I20260923 19:32:40.975467 14756 tablet_bootstrap.cc:654] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:40.976361 14757 tablet_bootstrap.cc:492] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: Bootstrap starting.
I20260923 19:32:40.977799 14758 tablet_bootstrap.cc:654] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:40.977792 14756 log.cc:824] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:40.978995 14758 log.cc:824] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:40.980357 14757 tablet_bootstrap.cc:654] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:40.981743 14756 tablet_bootstrap.cc:492] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: No bootstrap required, opened a new log
I20260923 19:32:40.981731 14757 log.cc:824] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:40.981877 14758 tablet_bootstrap.cc:492] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: No bootstrap required, opened a new log
I20260923 19:32:40.982076 14756 ts_tablet_manager.cc:1397] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 19:32:40.982141 14758 ts_tablet_manager.cc:1397] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 19:32:40.985142 14757 tablet_bootstrap.cc:492] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: No bootstrap required, opened a new log
I20260923 19:32:40.985440 14757 ts_tablet_manager.cc:1397] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260923 19:32:40.989553 14758 raft_consensus.cc:348] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } }
I20260923 19:32:40.989939 14758 raft_consensus.cc:374] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:40.990026 14758 raft_consensus.cc:729] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17532166823145ab87ea861bbc005d3f, State: Initialized, Role: FOLLOWER
I20260923 19:32:40.990779 14758 consensus_queue.cc:260] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } }
I20260923 19:32:40.992465 14737 heartbeater.cc:503] Master 127.0.97.62:43013 was elected leader, sending a full tablet report...
I20260923 19:32:40.993160 14758 ts_tablet_manager.cc:1428] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260923 19:32:40.993129 14756 raft_consensus.cc:348] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } }
I20260923 19:32:40.993678 14756 raft_consensus.cc:374] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:40.993789 14756 raft_consensus.cc:729] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1c8e4712f641288dde8ce6427a8a7e, State: Initialized, Role: FOLLOWER
I20260923 19:32:40.994612 14756 consensus_queue.cc:260] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } }
I20260923 19:32:40.996280 14757 raft_consensus.cc:348] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } }
I20260923 19:32:40.996766 14757 raft_consensus.cc:374] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:40.996867 14757 raft_consensus.cc:729] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b83ca6fbd643cb931810f78512ad66, State: Initialized, Role: FOLLOWER
I20260923 19:32:40.997287 14756 ts_tablet_manager.cc:1428] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260923 19:32:40.997659 14757 consensus_queue.cc:260] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } }
I20260923 19:32:40.999827 14757 ts_tablet_manager.cc:1428] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260923 19:32:40.999982 14603 heartbeater.cc:503] Master 127.0.97.62:43013 was elected leader, sending a full tablet report...
I20260923 19:32:41.020673 14764 raft_consensus.cc:482] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:32:41.021075 14764 raft_consensus.cc:504] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } }
W20260923 19:32:41.022994 14604 tablet.cc:2369] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:32:41.023329 14764 leader_election.cc:288] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f1c8e4712f641288dde8ce6427a8a7e (127.0.97.1:42795), 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093)
I20260923 19:32:41.032503 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:41.023932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39249 (local address 127.0.97.1:42795)
0923 19:32:41.024236 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:41.024243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:41.024893 (+ 650us) server_negotiation.cc:407] Connection header received
0923 19:32:41.025031 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:41.025037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:41.025130 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:41.025243 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:41.026342 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.027314 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.030624 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.031489 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.031627 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:41.031746 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:41.032113 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 19:32:41.032249 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260923 19:32:41.032621 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:41.024191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60155 (local address 127.0.97.3:42093)
0923 19:32:41.024423 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:41.024430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:41.024768 (+ 338us) server_negotiation.cc:407] Connection header received
0923 19:32:41.025031 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:41.025037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:41.025123 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:41.025243 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:41.026276 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.027255 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.030615 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.031535 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.031669 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:41.031798 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:41.032173 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 19:32:41.032322 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 19:32:41.032768 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:41.023704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.97.2:39249)
0923 19:32:41.024781 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:41.024797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:41.024911 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:41.025345 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:41.025352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:41.025389 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:41.026068 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:41.026084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.027442 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.027447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:41.030330 (+ 2883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:41.030354 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.031759 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.031767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:41.031945 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:41.031976 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 19:32:41.032069 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 19:32:41.032229 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":355,"thread_start_us":481,"threads_started":1}
I20260923 19:32:41.032820 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:41.023605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42093 (local address 127.0.97.2:60155)
0923 19:32:41.024610 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:41.024644 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:41.024859 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:41.025345 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:41.025352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:41.025476 (+ 124us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:41.026069 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:41.026084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.027411 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.027419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:41.030328 (+ 2909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:41.030354 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.031817 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.031825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:41.031998 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:41.032027 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 19:32:41.032132 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 19:32:41.032323 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":97,"spinlock_wait_cycles":28416,"thread_start_us":843,"threads_started":1}
I20260923 19:32:41.034128 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077d4a5da3d44c2fa9bd7ae47d5aea0d" candidate_uuid: "c4b83ca6fbd643cb931810f78512ad66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17532166823145ab87ea861bbc005d3f" is_pre_election: true
I20260923 19:32:41.034216 14424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077d4a5da3d44c2fa9bd7ae47d5aea0d" candidate_uuid: "c4b83ca6fbd643cb931810f78512ad66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" is_pre_election: true
I20260923 19:32:41.034781 14692 raft_consensus.cc:2454] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4b83ca6fbd643cb931810f78512ad66 in term 0.
I20260923 19:32:41.034875 14424 raft_consensus.cc:2454] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4b83ca6fbd643cb931810f78512ad66 in term 0.
I20260923 19:32:41.035583 14494 leader_election.cc:302] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17532166823145ab87ea861bbc005d3f, 8f1c8e4712f641288dde8ce6427a8a7e, c4b83ca6fbd643cb931810f78512ad66; no voters:
I20260923 19:32:41.036062 14764 raft_consensus.cc:2781] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:32:41.036240 14764 raft_consensus.cc:482] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:32:41.036355 14764 raft_consensus.cc:3037] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:41.038838 14764 raft_consensus.cc:504] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } }
I20260923 19:32:41.039569 14764 leader_election.cc:288] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [CANDIDATE]: Term 1 election: Requested vote from peers 8f1c8e4712f641288dde8ce6427a8a7e (127.0.97.1:42795), 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093)
I20260923 19:32:41.040169 14424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077d4a5da3d44c2fa9bd7ae47d5aea0d" candidate_uuid: "c4b83ca6fbd643cb931810f78512ad66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1c8e4712f641288dde8ce6427a8a7e"
I20260923 19:32:41.040409 14424 raft_consensus.cc:3037] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:41.040561 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077d4a5da3d44c2fa9bd7ae47d5aea0d" candidate_uuid: "c4b83ca6fbd643cb931810f78512ad66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17532166823145ab87ea861bbc005d3f"
I20260923 19:32:41.040822 14692 raft_consensus.cc:3037] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:41.042797 14424 raft_consensus.cc:2454] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4b83ca6fbd643cb931810f78512ad66 in term 1.
I20260923 19:32:41.043254 14491 leader_election.cc:302] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1c8e4712f641288dde8ce6427a8a7e, c4b83ca6fbd643cb931810f78512ad66; no voters:
I20260923 19:32:41.043612 14764 raft_consensus.cc:2781] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:32:41.043774 14692 raft_consensus.cc:2454] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4b83ca6fbd643cb931810f78512ad66 in term 1.
I20260923 19:32:41.044368 14764 raft_consensus.cc:686] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 1 LEADER]: Becoming Leader. State: Replica: c4b83ca6fbd643cb931810f78512ad66, State: Running, Role: LEADER
I20260923 19:32:41.045059 14764 consensus_queue.cc:237] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } }
I20260923 19:32:41.050802 14280 catalog_manager.cc:5495] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c4b83ca6fbd643cb931810f78512ad66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } health_report { overall_health: HEALTHY } } }
I20260923 19:32:41.096607 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:41.090740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.0.97.2:44133)
0923 19:32:41.091024 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:41.091031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:41.091053 (+ 22us) server_negotiation.cc:407] Connection header received
0923 19:32:41.091313 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:41.091319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:41.091423 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:41.091572 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:32:41.091975 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.092728 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.094130 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.094583 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.094689 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:41.095225 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:32:41.095887 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:32:41.096035 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:41.096126 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 19:32:41.096218 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260923 19:32:41.098111 14738 tablet.cc:2369] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:32:41.106914 14424 raft_consensus.cc:1261] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Refusing update from remote peer c4b83ca6fbd643cb931810f78512ad66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:32:41.106976 14692 raft_consensus.cc:1261] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [term 1 FOLLOWER]: Refusing update from remote peer c4b83ca6fbd643cb931810f78512ad66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:32:41.108068 14764 consensus_queue.cc:1035] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:32:41.108407 14767 consensus_queue.cc:1035] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:32:41.122036 14770 mvcc.cc:205] Tried to move back new op lower bound from 7332626272681197568 to 7332626272445747200. Current Snapshot: MvccSnapshot[applied={T|T < 7332626272681197568}]
I20260923 19:32:41.138135 14279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:32:41.138469 14279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:32:41.143200 14279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Sending DeleteTablet for 3 replicas of tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d
I20260923 19:32:41.144939 14538 tablet_service.cc:1514] Processing DeleteTablet for tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:32:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34216
I20260923 19:32:41.145063 14672 tablet_service.cc:1514] Processing DeleteTablet for tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:32:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52442
I20260923 19:32:41.145117 14404 tablet_service.cc:1514] Processing DeleteTablet for tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:32:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43774
I20260923 19:32:41.146102 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 14341
I20260923 19:32:41.155834 14780 tablet_replica.cc:330] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: stopping tablet replica
I20260923 19:32:41.156143 14779 tablet_replica.cc:330] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: stopping tablet replica
I20260923 19:32:41.156366 14780 raft_consensus.cc:2229] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:32:41.156702 14779 raft_consensus.cc:2229] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:32:41.156941 14780 raft_consensus.cc:2258] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:32:41.157680 14779 raft_consensus.cc:2258] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:32:41.157948 14781 tablet_replica.cc:330] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: stopping tablet replica
I20260923 19:32:41.158870 14781 raft_consensus.cc:2229] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:32:41.159377 14781 raft_consensus.cc:2258] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:32:41.160266 14779 ts_tablet_manager.cc:1905] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:32:41.161895 14781 ts_tablet_manager.cc:1905] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:32:41.162915 14780 ts_tablet_manager.cc:1905] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:32:41.169358 14781 ts_tablet_manager.cc:1918] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:32:41.169531 14781 log.cc:1196] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/077d4a5da3d44c2fa9bd7ae47d5aea0d
I20260923 19:32:41.170017 14781 ts_tablet_manager.cc:1939] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 17532166823145ab87ea861bbc005d3f: Deleting consensus metadata
I20260923 19:32:41.172340 14267 catalog_manager.cc:4841] TS 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093): tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d (table test-table [id=7b90b728ba984d4fbb9039787da67ad4]) successfully deleted
I20260923 19:32:41.174629 14779 ts_tablet_manager.cc:1918] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 23 19:32:41 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:32:41.174952 14779 log.cc:1196] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/077d4a5da3d44c2fa9bd7ae47d5aea0d
I20260923 19:32:41.175647 14779 ts_tablet_manager.cc:1939] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P c4b83ca6fbd643cb931810f78512ad66: Deleting consensus metadata
I20260923 19:32:41.176532 14780 ts_tablet_manager.cc:1918] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:32:41.176787 14780 log.cc:1196] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/077d4a5da3d44c2fa9bd7ae47d5aea0d
I20260923 19:32:41.177403 14780 ts_tablet_manager.cc:1939] T 077d4a5da3d44c2fa9bd7ae47d5aea0d P 8f1c8e4712f641288dde8ce6427a8a7e: Deleting consensus metadata
I20260923 19:32:41.177769 14264 catalog_manager.cc:4841] TS c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133): tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d (table test-table [id=7b90b728ba984d4fbb9039787da67ad4]) successfully deleted
I20260923 19:32:41.179767 14264 catalog_manager.cc:4841] TS 8f1c8e4712f641288dde8ce6427a8a7e (127.0.97.1:42795): tablet 077d4a5da3d44c2fa9bd7ae47d5aea0d (table test-table [id=7b90b728ba984d4fbb9039787da67ad4]) successfully deleted
I20260923 19:32:41.182960 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:41.147505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.0.97.1:42795)
0923 19:32:41.147715 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:41.147722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:41.163254 (+ 15532us) server_negotiation.cc:407] Connection header received
0923 19:32:41.163605 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:41.163612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:41.163718 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:41.163809 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:41.165920 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.167123 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.169242 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.169914 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.170015 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:41.176345 (+ 6330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:41.176365 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:41.176383 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:41.176435 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:41.180104 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:41.180816 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:41.180823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:41.180825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:41.180950 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:41.181334 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:41.181337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:41.181339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:41.181744 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:41.182018 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:41.182386 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 19:32:41.182612 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 19:32:41.183665 14464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:41.276893 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 14341
I20260923 19:32:41.293491 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 14475
Sep 23 19:32:41 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:32:41.312445 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:41.294470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.0.97.2:44133)
0923 19:32:41.294675 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:41.294682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:41.294738 (+ 56us) server_negotiation.cc:407] Connection header received
0923 19:32:41.294948 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:41.294952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:41.295035 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:41.295155 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:41.296444 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.297605 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.299948 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.300698 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.300874 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:41.306046 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:41.306065 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:41.306081 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:41.306126 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:41.309307 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:41.310101 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:41.310108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:41.310110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:41.310264 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:41.310811 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:41.310814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:41.310817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:41.311202 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:41.311533 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:41.311791 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 19:32:41.312046 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 19:32:41.313675 14598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:41.432904 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 14475
I20260923 19:32:41.448880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 14609
Sep 23 19:32:41 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:32:41.467401 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:41.449678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.0.97.3:42093)
0923 19:32:41.449845 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:41.449849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:41.449950 (+ 101us) server_negotiation.cc:407] Connection header received
0923 19:32:41.450168 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:41.450172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:41.450241 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:41.450386 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:41.451863 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.452706 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.454161 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:41.454590 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:41.454693 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:41.459594 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:41.459615 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:41.459635 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:41.459690 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:41.463024 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:41.463981 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:41.463993 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:41.463996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:41.464187 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:41.464766 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:41.464772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:41.464775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:41.465358 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:41.465654 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:41.466528 (+ 874us) server_negotiation.cc:299] Negotiation successful
0923 19:32:41.466970 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 19:32:41.468317 14732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:41.565856 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 14609
I20260923 19:32:41.581960 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 14248
I20260923 19:32:41.583411 14310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:41.707717 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 14248
I20260923 19:32:41.727905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--webserver_interface=127.0.97.62
--webserver_port=39987
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:42.060464 14787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:42.060822 14787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:42.060961 14787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:42.068287 14787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:32:42.068428 14787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:42.068503 14787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:42.068583 14787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:32:42.068646 14787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:32:42.078676 14787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:42.080001 14787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:42.081394 14787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:42.105901 14796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:42.105902 14794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:42.105901 14793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:42.107002 14787 server_base.cc:1034] running on GCE node
I20260923 19:32:42.108088 14787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:42.110131 14787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:42.111399 14787 hybrid_clock.cc:648] HybridClock initialized: now 1790191962111329 us; error 56 us; skew 500 ppm
I20260923 19:32:42.111891 14787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:42 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:42.115919 14787 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:32:42.118014 14787 webserver.cc:460] Webserver started at http://127.0.97.62:39987/ using document root <none> and password file <none>
I20260923 19:32:42.118973 14787 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:42.119113 14787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:42.127154 14787 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 19:32:42.131196 14803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:42.133137 14787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:32:42.133350 14787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "8134fe0d94a64de6a110832539a4f405"
format_stamp: "Formatted at 2026-09-23 19:32:38 on dist-test-slave-kxfb"
server_key: "b97fba90af9299ed56090554b9b6476c"
server_key_iv: "799a1693aae4c8baffb3c9493763a605"
server_key_version: "encryptionkey@0"
I20260923 19:32:42.134696 14787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:42.166316 14787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:42.171375 14787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:42.171692 14787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:42.192832 14787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43013
I20260923 19:32:42.192862 14854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43013 every 8 connection(s)
I20260923 19:32:42.194752 14787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:32:42.202333 14855 sys_catalog.cc:263] Verifying existing consensus state
I20260923 19:32:42.203738 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 14787
I20260923 19:32:42.204617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42795
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43013
--webserver_port=37931
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:32:42.206038 14855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Bootstrap starting.
I20260923 19:32:42.226234 14855 log.cc:824] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:42.239392 14855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 19:32:42.240012 14855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Bootstrap complete.
I20260923 19:32:42.249302 14855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:42.250214 14855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8134fe0d94a64de6a110832539a4f405, State: Initialized, Role: FOLLOWER
I20260923 19:32:42.250913 14855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:42.251152 14855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:32:42.251261 14855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:32:42.251420 14855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 1 FOLLOWER]: Advancing to term 2
I20260923 19:32:42.255455 14855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:42.255972 14855 leader_election.cc:302] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8134fe0d94a64de6a110832539a4f405; no voters:
I20260923 19:32:42.256608 14855 leader_election.cc:288] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 19:32:42.256752 14859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 19:32:42.258273 14859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 2 LEADER]: Becoming Leader. State: Replica: 8134fe0d94a64de6a110832539a4f405, State: Running, Role: LEADER
I20260923 19:32:42.259014 14859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:42.259289 14855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:32:42.262007 14861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8134fe0d94a64de6a110832539a4f405. Latest consensus state: current_term: 2 leader_uuid: "8134fe0d94a64de6a110832539a4f405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260923 19:32:42.262624 14861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:42.263445 14860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8134fe0d94a64de6a110832539a4f405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260923 19:32:42.263756 14860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:42.271119 14866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:32:42.280942 14866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:32:42.282857 14866 catalog_manager.cc:1259] Loaded cluster ID: 76b60dec10ff4e8488346fbbfe9f6e2e
I20260923 19:32:42.283042 14866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:32:42.286981 14866 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 19:32:42.310106 14866 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 19:32:42.319806 14866 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:32:42.321372 14866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Loaded TSK: 0
I20260923 19:32:42.322774 14866 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 19:32:42.589440 14857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:42.589735 14857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:42.589854 14857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:42.597222 14857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:42.597347 14857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:42.597482 14857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:32:42.607756 14857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37931
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:42.609093 14857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:42.610466 14857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:42.624053 14883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:42.624043 14882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:42.625110 14885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:42.625147 14857 server_base.cc:1034] running on GCE node
I20260923 19:32:42.626217 14857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:42.627866 14857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:42.629153 14857 hybrid_clock.cc:648] HybridClock initialized: now 1790191962629085 us; error 69 us; skew 500 ppm
I20260923 19:32:42.629643 14857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:42 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:42.635113 14857 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:32:42.637328 14857 webserver.cc:460] Webserver started at http://127.0.97.1:37931/ using document root <none> and password file <none>
I20260923 19:32:42.638351 14857 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:42.638525 14857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:42.646437 14857 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 19:32:42.650343 14892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:42.652313 14857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:32:42.652521 14857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "8f1c8e4712f641288dde8ce6427a8a7e"
format_stamp: "Formatted at 2026-09-23 19:32:39 on dist-test-slave-kxfb"
server_key: "d75a6a2ccec94d36cd0a3e50e37ccd32"
server_key_iv: "29bc918032cfdfae9bef4021a6450beb"
server_key_version: "encryptionkey@0"
I20260923 19:32:42.653971 14857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:42.671512 14857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:42.676954 14857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:42.677268 14857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:42.678601 14857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:42.680411 14857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:42.680522 14857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:42.680696 14857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:42.680783 14857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:42.721352 14857 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42795
I20260923 19:32:42.721503 15004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42795 every 8 connection(s)
I20260923 19:32:42.723177 14857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:32:42.732932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 14857
I20260923 19:32:42.733551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44133
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43013
--webserver_port=37731
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:32:42 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:42.752323 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:42.726850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50577 (local address 127.0.97.62:43013)
0923 19:32:42.727679 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:42.727702 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:42.728385 (+ 683us) server_negotiation.cc:407] Connection header received
0923 19:32:42.730918 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:42.730932 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:42.731364 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:42.731735 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:42.733659 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:42.735113 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:42.736791 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:42.737492 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:42.737696 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:42.742474 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:42.742490 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:42.742517 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:42.742587 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:42.746619 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:42.747654 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:42.747663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:42.747673 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:42.747916 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:42.748413 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:42.748418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:42.748421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:42.749123 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:42.749331 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:42.750402 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0923 19:32:42.750911 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":395,"threads_started":1}
I20260923 19:32:42.752321 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:42.727098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:50577)
0923 19:32:42.728178 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:42.728226 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:42.730233 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:42.731939 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:42.731950 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:42.732723 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:42.733397 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:42.733427 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:42.735292 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:42.735297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:42.736627 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:42.736643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:42.737756 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:42.737764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:42.738069 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:42.738903 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:42.738932 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:42.741984 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:42.746866 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:42.746878 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:42.746896 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:42.747479 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:42.748067 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:42.748071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:42.748073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:42.748291 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:42.749284 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:42.749291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:42.749715 (+ 424us) client_negotiation.cc:769] Sending connection context
0923 19:32:42.750026 (+ 311us) client_negotiation.cc:240] Negotiation successful
0923 19:32:42.750751 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":479,"threads_started":1}
I20260923 19:32:42.756151 15005 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43013
I20260923 19:32:42.756505 15005 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:42.757375 15005 heartbeater.cc:511] Master 127.0.97.62:43013 requested a full tablet report, sending...
I20260923 19:32:42.760134 14820 ts_manager.cc:194] Registered new tserver with Master: 8f1c8e4712f641288dde8ce6427a8a7e (127.0.97.1:42795)
I20260923 19:32:42.762420 14820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50577
W20260923 19:32:43.062505 15009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:43.062803 15009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:43.062897 15009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:43.070238 15009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:43.070386 15009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:43.070506 15009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:32:43.080443 15009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37731
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:43.081780 15009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:43.083107 15009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:43.097388 15017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:43.097415 15014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:43.097936 15015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:43.098840 15009 server_base.cc:1034] running on GCE node
I20260923 19:32:43.099479 15009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:43.100509 15009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:43.101725 15009 hybrid_clock.cc:648] HybridClock initialized: now 1790191963101654 us; error 62 us; skew 500 ppm
I20260923 19:32:43.102062 15009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:43 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:43.105595 15009 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:32:43.107402 15009 webserver.cc:460] Webserver started at http://127.0.97.2:37731/ using document root <none> and password file <none>
I20260923 19:32:43.108108 15009 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:43.108245 15009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:43.113862 15009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:32:43.116892 15024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:43.118500 15009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 19:32:43.118674 15009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "c4b83ca6fbd643cb931810f78512ad66"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "caa2afe31adcd2e534e3cb4da8b104e7"
server_key_iv: "a6b86d66706b897343d589d388a2444c"
server_key_version: "encryptionkey@0"
I20260923 19:32:43.119655 15009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:43.155198 15009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:43.159356 15009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:43.159597 15009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:43.160857 15009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:43.162359 15009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:43.162452 15009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:43.162540 15009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:43.162601 15009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:43.198037 15009 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44133
I20260923 19:32:43.198091 15136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44133 every 8 connection(s)
I20260923 19:32:43.199512 15009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:32:43.209764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 15009
I20260923 19:32:43.210497 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42093
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43013
--webserver_port=42311
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:32:43 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:43.225406 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.202856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49601 (local address 127.0.97.62:43013)
0923 19:32:43.203198 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.203206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.204506 (+ 1300us) server_negotiation.cc:407] Connection header received
0923 19:32:43.207520 (+ 3014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.207525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.207666 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.207783 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:43.210408 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.211506 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.213668 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.214365 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.214517 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.218757 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:43.218762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:43.218772 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:43.218812 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:43.221526 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:43.222433 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:43.222438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:43.222442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:43.222579 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:43.223174 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:43.223183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:43.223186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:43.223670 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:43.223852 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.224694 (+ 842us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.225012 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260923 19:32:43.226176 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.203322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:49601)
0923 19:32:43.204268 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:43.204325 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:43.207099 (+ 2774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:43.208176 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:43.208191 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:43.209124 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:43.210106 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.210135 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.211738 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.211746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.213471 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.213487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.214628 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.214634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.214869 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.215687 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:43.215719 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:43.218479 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:43.221731 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:43.221741 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:43.221755 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:43.222262 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:43.222692 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:43.222698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:43.222701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:43.222926 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:43.223829 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:43.223834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:43.224241 (+ 407us) client_negotiation.cc:769] Sending connection context
0923 19:32:43.224530 (+ 289us) client_negotiation.cc:240] Negotiation successful
0923 19:32:43.224977 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":344,"threads_started":1}
I20260923 19:32:43.227982 15137 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43013
I20260923 19:32:43.228493 15137 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:43.229657 15137 heartbeater.cc:511] Master 127.0.97.62:43013 requested a full tablet report, sending...
I20260923 19:32:43.231700 14820 ts_manager.cc:194] Registered new tserver with Master: c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133)
I20260923 19:32:43.232537 14820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49601
W20260923 19:32:43.585355 15140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:43.585700 15140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:43.585857 15140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:43.593638 15140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:43.593780 15140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:43.593951 15140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:32:43.604560 15140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42311
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:43.606338 15140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:43.607959 15140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:43.622889 15145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:43.623032 15148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:43.623045 15146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:43.624233 15140 server_base.cc:1034] running on GCE node
I20260923 19:32:43.624972 15140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:43.626152 15140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:43.627399 15140 hybrid_clock.cc:648] HybridClock initialized: now 1790191963627333 us; error 58 us; skew 500 ppm
I20260923 19:32:43.627748 15140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:43 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:43.631078 15140 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:32:43.632718 15140 webserver.cc:460] Webserver started at http://127.0.97.3:42311/ using document root <none> and password file <none>
I20260923 19:32:43.633443 15140 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:43.633560 15140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:43.639611 15140 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:32:43.642786 15155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:43.644354 15140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 19:32:43.644517 15140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "17532166823145ab87ea861bbc005d3f"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "cfa0b0bb54a1e445316cdc6e1f27de13"
server_key_iv: "bc93b09e212e680de78f0360326869e6"
server_key_version: "encryptionkey@0"
I20260923 19:32:43.645533 15140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:43.664672 15140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:43.669068 15140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:43.669298 15140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:43.670378 15140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:43.671803 15140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:43.671896 15140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:43.671984 15140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:43.672083 15140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:43.705636 15140 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42093
I20260923 19:32:43.705693 15267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42093 every 8 connection(s)
I20260923 19:32:43.707314 15140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:32:43.717505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 15140
Sep 23 19:32:43 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:32:43.731793 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.710596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34839 (local address 127.0.97.62:43013)
0923 19:32:43.710874 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.710879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.712414 (+ 1535us) server_negotiation.cc:407] Connection header received
0923 19:32:43.714441 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.714446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.714539 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.714662 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:43.716370 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.717228 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.719418 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.720062 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.720170 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.724686 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:43.724694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:43.724699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:43.724741 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:43.728041 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:43.728797 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:43.728801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:43.728803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:43.728954 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:43.729609 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:43.729613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:43.729615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:43.729962 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:43.730127 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.731137 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.731400 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260923 19:32:43.732690 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.711199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:34839)
0923 19:32:43.712196 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:43.712255 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:43.714047 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:43.714914 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:43.714924 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:43.715526 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:43.716168 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.716187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.717474 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.717479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.719239 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.719259 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.720339 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.720375 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.720741 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.721649 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:43.721680 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:43.724404 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:43.728180 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:43.728192 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:43.728213 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:43.728667 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:43.729186 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:43.729191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:43.729194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:43.729462 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:43.730155 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:43.730165 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:43.730616 (+ 451us) client_negotiation.cc:769] Sending connection context
0923 19:32:43.730935 (+ 319us) client_negotiation.cc:240] Negotiation successful
0923 19:32:43.731417 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":382,"threads_started":1}
I20260923 19:32:43.734655 15268 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43013
I20260923 19:32:43.735008 15268 heartbeater.cc:464] Registering TS with master...
I20260923 19:32:43.735829 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.718527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.0.97.62:43013)
0923 19:32:43.719304 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.719308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.719328 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:32:43.719470 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.719474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.719562 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.719695 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:43.721693 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.722856 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.725322 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.725999 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.726141 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.729000 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:43.729006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:43.729011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:43.729047 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:43.732205 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:43.733460 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:43.733465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:43.733469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:43.733608 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:43.734092 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:43.734100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:43.734103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:43.734551 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:43.734833 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.735176 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.735441 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":549,"threads_started":1}
I20260923 19:32:43.735960 15268 heartbeater.cc:511] Master 127.0.97.62:43013 requested a full tablet report, sending...
I20260923 19:32:43.737717 14819 ts_manager.cc:194] Registered new tserver with Master: 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093)
I20260923 19:32:43.738852 14819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34839
I20260923 19:32:43.751375 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:32:43.759086 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.752437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.0.97.62:43013)
0923 19:32:43.752632 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.752637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.753171 (+ 534us) server_negotiation.cc:407] Connection header received
0923 19:32:43.753450 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.753454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.753614 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.753761 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:32:43.754223 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.754991 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.756599 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.757016 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.757135 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.757692 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:32:43.758442 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:32:43.758571 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.758765 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.758856 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 19:32:43.766392 15005 heartbeater.cc:503] Master 127.0.97.62:43013 was elected leader, sending a full tablet report...
I20260923 19:32:43.760402 14819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:32:43.769187 14819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:32:43.794026 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.783180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.0.97.2:44133)
0923 19:32:43.783788 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.783798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.784507 (+ 709us) server_negotiation.cc:407] Connection header received
0923 19:32:43.784893 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.784901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.785184 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.785355 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:43.786765 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.788163 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.791112 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.792636 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.792772 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.792973 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.793560 (+ 587us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.793723 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":267,"threads_started":1}
I20260923 19:32:43.794026 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.783392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.0.97.3:42093)
0923 19:32:43.784032 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.784041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.784588 (+ 547us) server_negotiation.cc:407] Connection header received
0923 19:32:43.784951 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.784961 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.785242 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.785437 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:43.787006 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.788501 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.790910 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.792272 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.792422 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.792697 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.793465 (+ 768us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.793689 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":342,"threads_started":1}
I20260923 19:32:43.794111 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.782808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42093 (local address 127.0.0.1:52450)
0923 19:32:43.784358 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:43.784401 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:43.784611 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:43.785677 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:43.785685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:43.785715 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:43.786759 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.786772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.788795 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.788800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.790706 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.790721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.792844 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.792849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.793079 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.793206 (+ 127us) client_negotiation.cc:769] Sending connection context
0923 19:32:43.793380 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 19:32:43.793636 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"mutex_wait_us":482,"thread_start_us":1122,"threads_started":1}
I20260923 19:32:43.794971 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.783222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44133 (local address 127.0.0.1:34228)
0923 19:32:43.784354 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:43.784398 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:43.784620 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:43.785803 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:43.785812 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:43.785869 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:43.786520 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.786535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.788335 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.788340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.790916 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.790929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.793117 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.793121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.793318 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.793352 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:32:43.793441 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 19:32:43.794435 (+ 994us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":817,"spinlock_wait_cycles":11264,"thread_start_us":1404,"threads_started":1}
I20260923 19:32:43.795784 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.785029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43778)
0923 19:32:43.786662 (+ 1633us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:43.786691 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:43.786841 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:43.787465 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:43.787469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:43.787495 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:43.788120 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.788135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.790516 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.790521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.792769 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.792788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.795011 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.795019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.795217 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.795250 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 19:32:43.795339 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 19:32:43.795484 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1479,"thread_start_us":1286,"threads_started":1}
I20260923 19:32:43.796841 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.785632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.0.97.1:42795)
0923 19:32:43.786472 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.786483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.786825 (+ 342us) server_negotiation.cc:407] Connection header received
0923 19:32:43.787006 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.787013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.787262 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.787472 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:43.788695 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.790349 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.792948 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.794446 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.794575 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.794801 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.796316 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.796500 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":391,"threads_started":1}
I20260923 19:32:43.798714 15072 tablet_service.cc:1467] Processing CreateTablet for tablet fa8de6a4ac4a4d06908d9c423e0f9d38 (DEFAULT_TABLE table=test-table [id=afa973c7000e4173b27e3680a5c33324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:32:43.799237 15203 tablet_service.cc:1467] Processing CreateTablet for tablet fa8de6a4ac4a4d06908d9c423e0f9d38 (DEFAULT_TABLE table=test-table [id=afa973c7000e4173b27e3680a5c33324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:32:43.799913 15072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8de6a4ac4a4d06908d9c423e0f9d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:43.800886 15203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8de6a4ac4a4d06908d9c423e0f9d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:43.800474 14940 tablet_service.cc:1467] Processing CreateTablet for tablet fa8de6a4ac4a4d06908d9c423e0f9d38 (DEFAULT_TABLE table=test-table [id=afa973c7000e4173b27e3680a5c33324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:32:43.801828 14940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8de6a4ac4a4d06908d9c423e0f9d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:32:43.812175 15284 tablet_bootstrap.cc:492] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: Bootstrap starting.
I20260923 19:32:43.816015 15284 tablet_bootstrap.cc:654] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:43.817400 15284 log.cc:824] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:43.818627 15286 tablet_bootstrap.cc:492] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: Bootstrap starting.
I20260923 19:32:43.819679 15285 tablet_bootstrap.cc:492] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Bootstrap starting.
I20260923 19:32:43.823233 15284 tablet_bootstrap.cc:492] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: No bootstrap required, opened a new log
I20260923 19:32:43.823606 15284 ts_tablet_manager.cc:1397] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260923 19:32:43.823679 15286 tablet_bootstrap.cc:654] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:43.823911 15285 tablet_bootstrap.cc:654] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Neither blocks nor log segments found. Creating new log.
I20260923 19:32:43.825073 15286 log.cc:824] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:43.825127 15285 log.cc:824] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:43.828605 15285 tablet_bootstrap.cc:492] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: No bootstrap required, opened a new log
I20260923 19:32:43.828624 15286 tablet_bootstrap.cc:492] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: No bootstrap required, opened a new log
I20260923 19:32:43.828905 15286 ts_tablet_manager.cc:1397] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260923 19:32:43.828909 15285 ts_tablet_manager.cc:1397] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 19:32:43.832494 15284 raft_consensus.cc:348] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:43.832998 15284 raft_consensus.cc:374] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:43.833112 15284 raft_consensus.cc:729] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b83ca6fbd643cb931810f78512ad66, State: Initialized, Role: FOLLOWER
I20260923 19:32:43.833775 15284 consensus_queue.cc:260] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:43.835004 15137 heartbeater.cc:503] Master 127.0.97.62:43013 was elected leader, sending a full tablet report...
I20260923 19:32:43.837200 15284 ts_tablet_manager.cc:1428] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260923 19:32:43.839574 15286 raft_consensus.cc:348] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:43.839592 15285 raft_consensus.cc:348] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:43.840085 15286 raft_consensus.cc:374] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:43.840085 15285 raft_consensus.cc:374] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:32:43.840198 15285 raft_consensus.cc:729] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1c8e4712f641288dde8ce6427a8a7e, State: Initialized, Role: FOLLOWER
I20260923 19:32:43.840202 15286 raft_consensus.cc:729] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17532166823145ab87ea861bbc005d3f, State: Initialized, Role: FOLLOWER
I20260923 19:32:43.840886 15286 consensus_queue.cc:260] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:43.840886 15285 consensus_queue.cc:260] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:43.842970 15268 heartbeater.cc:503] Master 127.0.97.62:43013 was elected leader, sending a full tablet report...
I20260923 19:32:43.843001 15285 ts_tablet_manager.cc:1428] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 19:32:43.843282 15286 ts_tablet_manager.cc:1428] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260923 19:32:43.846040 15291 raft_consensus.cc:482] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:32:43.846356 15291 raft_consensus.cc:504] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:43.848209 15291 leader_election.cc:288] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133), 8f1c8e4712f641288dde8ce6427a8a7e (127.0.97.1:42795)
I20260923 19:32:43.854717 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.848527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44133 (local address 127.0.97.3:55711)
0923 19:32:43.848783 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:43.848802 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:43.848916 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:43.849702 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:43.849706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:43.849732 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:43.849939 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.849950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.851044 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.851049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.852371 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.852383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.853706 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.853714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.854026 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.854071 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 19:32:43.854183 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 19:32:43.854398 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260923 19:32:43.855971 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.849435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42009 (local address 127.0.97.1:42795)
0923 19:32:43.849633 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.849640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.849725 (+ 85us) server_negotiation.cc:407] Connection header received
0923 19:32:43.850031 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.850036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.850122 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.850194 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:43.850762 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.851542 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.853495 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.854736 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.854886 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.854988 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.855571 (+ 583us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.855726 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 19:32:43.856844 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.848916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.97.3:42009)
0923 19:32:43.849637 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:43.849657 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:43.849899 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:43.850222 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:43.850226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:43.850251 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:32:43.850624 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.850633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.851693 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.851698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.853250 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:43.853264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.855168 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.855174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:43.855371 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.855408 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 19:32:43.856244 (+ 836us) client_negotiation.cc:240] Negotiation successful
0923 19:32:43.856424 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"spinlock_wait_cycles":28416,"thread_start_us":402,"threads_started":1}
I20260923 19:32:43.857381 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.848699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55711 (local address 127.0.97.2:44133)
0923 19:32:43.849401 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.849407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.849427 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:32:43.849501 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.849507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.849578 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.849662 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:32:43.850070 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.850910 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.852523 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.853505 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.853640 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.853740 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.854350 (+ 610us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.854645 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521}
I20260923 19:32:43.858084 14960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8de6a4ac4a4d06908d9c423e0f9d38" candidate_uuid: "17532166823145ab87ea861bbc005d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" is_pre_election: true
I20260923 19:32:43.858506 15092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8de6a4ac4a4d06908d9c423e0f9d38" candidate_uuid: "17532166823145ab87ea861bbc005d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b83ca6fbd643cb931810f78512ad66" is_pre_election: true
I20260923 19:32:43.858752 14960 raft_consensus.cc:2454] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17532166823145ab87ea861bbc005d3f in term 0.
I20260923 19:32:43.859010 15092 raft_consensus.cc:2454] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17532166823145ab87ea861bbc005d3f in term 0.
I20260923 19:32:43.859577 15156 leader_election.cc:302] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17532166823145ab87ea861bbc005d3f, 8f1c8e4712f641288dde8ce6427a8a7e; no voters:
I20260923 19:32:43.859948 15291 raft_consensus.cc:2781] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:32:43.860112 15291 raft_consensus.cc:482] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:32:43.860227 15291 raft_consensus.cc:3037] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:43.862540 15291 raft_consensus.cc:504] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:43.863147 15291 leader_election.cc:288] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [CANDIDATE]: Term 1 election: Requested vote from peers c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133), 8f1c8e4712f641288dde8ce6427a8a7e (127.0.97.1:42795)
I20260923 19:32:43.863654 15092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8de6a4ac4a4d06908d9c423e0f9d38" candidate_uuid: "17532166823145ab87ea861bbc005d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b83ca6fbd643cb931810f78512ad66"
I20260923 19:32:43.863716 14960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8de6a4ac4a4d06908d9c423e0f9d38" candidate_uuid: "17532166823145ab87ea861bbc005d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1c8e4712f641288dde8ce6427a8a7e"
I20260923 19:32:43.863837 15092 raft_consensus.cc:3037] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:43.863986 14960 raft_consensus.cc:3037] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:32:43.866142 15092 raft_consensus.cc:2454] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17532166823145ab87ea861bbc005d3f in term 1.
I20260923 19:32:43.866622 15156 leader_election.cc:302] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17532166823145ab87ea861bbc005d3f, c4b83ca6fbd643cb931810f78512ad66; no voters:
I20260923 19:32:43.866607 14960 raft_consensus.cc:2454] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17532166823145ab87ea861bbc005d3f in term 1.
I20260923 19:32:43.866962 15291 raft_consensus.cc:2781] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:32:43.867532 15291 raft_consensus.cc:686] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 1 LEADER]: Becoming Leader. State: Replica: 17532166823145ab87ea861bbc005d3f, State: Running, Role: LEADER
I20260923 19:32:43.868088 15291 consensus_queue.cc:237] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:43.873531 14819 catalog_manager.cc:5495] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17532166823145ab87ea861bbc005d3f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "17532166823145ab87ea861bbc005d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } health_report { overall_health: UNKNOWN } } }
I20260923 19:32:43.907012 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.900988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.0.97.3:42093)
0923 19:32:43.901172 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.901176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.901197 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:32:43.901294 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.901299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.901379 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.901538 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:32:43.901993 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.902814 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.904538 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.904974 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.905090 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.905679 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:32:43.906333 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:32:43.906475 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.906634 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.906726 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 19:32:43.916626 15092 raft_consensus.cc:1261] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [term 1 FOLLOWER]: Refusing update from remote peer 17532166823145ab87ea861bbc005d3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:32:43.916621 14960 raft_consensus.cc:1261] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Refusing update from remote peer 17532166823145ab87ea861bbc005d3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:32:43.917685 15294 consensus_queue.cc:1035] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:32:43.918113 15291 consensus_queue.cc:1035] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:32:43.948359 14819 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:32:43.948678 14819 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:32:43.953109 14819 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Sending DeleteTablet for 3 replicas of tablet fa8de6a4ac4a4d06908d9c423e0f9d38
I20260923 19:32:43.954439 15203 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8de6a4ac4a4d06908d9c423e0f9d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:32:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52450
I20260923 19:32:43.954676 15072 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8de6a4ac4a4d06908d9c423e0f9d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:32:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34228
I20260923 19:32:43.955228 14940 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8de6a4ac4a4d06908d9c423e0f9d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:32:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43778
I20260923 19:32:43.955850 15308 tablet_replica.cc:330] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: stopping tablet replica
I20260923 19:32:43.956453 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 14857
I20260923 19:32:43.956442 15308 raft_consensus.cc:2229] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:32:43.957185 15308 raft_consensus.cc:2258] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:32:43.960090 15309 tablet_replica.cc:330] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: stopping tablet replica
I20260923 19:32:43.960081 15308 ts_tablet_manager.cc:1905] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:32:43.960599 15309 raft_consensus.cc:2229] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:32:43.963017 15309 raft_consensus.cc:2258] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:32:43.966511 15309 ts_tablet_manager.cc:1905] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:32:43.968477 15307 tablet_replica.cc:330] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: stopping tablet replica
I20260923 19:32:43.969035 15307 raft_consensus.cc:2229] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:32:43.969810 15307 raft_consensus.cc:2258] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:32:43.969897 15308 ts_tablet_manager.cc:1918] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:32:43.970162 15308 log.cc:1196] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/fa8de6a4ac4a4d06908d9c423e0f9d38
I20260923 19:32:43.970882 15308 ts_tablet_manager.cc:1939] T fa8de6a4ac4a4d06908d9c423e0f9d38 P c4b83ca6fbd643cb931810f78512ad66: Deleting consensus metadata
I20260923 19:32:43.972435 15307 ts_tablet_manager.cc:1905] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:32:43.973223 14804 catalog_manager.cc:4841] TS c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133): tablet fa8de6a4ac4a4d06908d9c423e0f9d38 (table test-table [id=afa973c7000e4173b27e3680a5c33324]) successfully deleted
I20260923 19:32:43.978068 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:43.957329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.0.97.1:42795)
0923 19:32:43.957525 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:43.957531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:43.957611 (+ 80us) server_negotiation.cc:407] Connection header received
0923 19:32:43.957810 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:43.957816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:43.957903 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:43.958019 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:43.959760 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.960913 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.962753 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:43.963486 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:43.963598 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:43.970147 (+ 6549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:43.970164 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:43.970183 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:43.970232 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:43.974205 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:43.975398 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:43.975408 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:43.975411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:43.975591 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:43.976031 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:43.976036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:43.976039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:43.976531 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:43.976863 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:43.977229 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 19:32:43.977472 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 19:32:43.978920 15000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:43.979574 15307 ts_tablet_manager.cc:1918] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:32:43.979781 15307 log.cc:1196] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/fa8de6a4ac4a4d06908d9c423e0f9d38
I20260923 19:32:43.980440 15307 ts_tablet_manager.cc:1939] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 17532166823145ab87ea861bbc005d3f: Deleting consensus metadata
I20260923 19:32:43.982780 14807 catalog_manager.cc:4841] TS 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093): tablet fa8de6a4ac4a4d06908d9c423e0f9d38 (table test-table [id=afa973c7000e4173b27e3680a5c33324]) successfully deleted
I20260923 19:32:44.136554 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 14857
I20260923 19:32:44.155696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 15009
W20260923 19:32:44.155686 14804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42795 (error 108)
I20260923 19:32:44.158128 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.157565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43800)
0923 19:32:44.157785 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:44.157928 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 19:32:44.158514 14804 catalog_manager.cc:4584] TS 8f1c8e4712f641288dde8ce6427a8a7e (127.0.97.1:42795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa8de6a4ac4a4d06908d9c423e0f9d38: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42795: connect: Connection refused (error 111)
I20260923 19:32:44.173588 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.156738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.97.2:44133)
0923 19:32:44.156970 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:44.156976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:44.157006 (+ 30us) server_negotiation.cc:407] Connection header received
0923 19:32:44.157216 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:44.157223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:44.157357 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:44.157488 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:44.159484 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:44.160356 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:44.162515 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:44.162967 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:44.163082 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:44.166781 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:44.166799 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:44.166818 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:44.166861 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:44.169925 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:44.170996 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:44.171005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:44.171007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:44.171153 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:44.171611 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:44.171615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:44.171617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:44.172140 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:44.172431 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:44.172901 (+ 470us) server_negotiation.cc:299] Negotiation successful
0923 19:32:44.173139 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 19:32:44.174472 15132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:44.212602 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.211850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43810)
0923 19:32:44.212230 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:44.212362 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260923 19:32:44.214185 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.213653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43826)
0923 19:32:44.213893 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:44.214019 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260923 19:32:44.275420 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.274838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43838)
0923 19:32:44.275075 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:44.275183 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 19:32:44.276602 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.276239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43844)
0923 19:32:44.276409 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:44.276486 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 19:32:44.324455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 15009
I20260923 19:32:44.346698 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 15140
I20260923 19:32:44.357906 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.357242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43846)
0923 19:32:44.357578 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:44.357688 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260923 19:32:44.359637 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.358991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43848)
0923 19:32:44.359310 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:44.359442 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260923 19:32:44.365934 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.347719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.0.97.3:42093)
0923 19:32:44.347946 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:44.347952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:44.347976 (+ 24us) server_negotiation.cc:407] Connection header received
0923 19:32:44.348191 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:44.348197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:44.348302 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:44.348448 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:44.350644 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:44.351673 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:44.353608 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:44.354176 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:44.354378 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:44.358313 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:44.358345 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:44.358373 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:44.358438 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:44.362539 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:44.363558 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:44.363565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:44.363572 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:44.363766 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:44.364208 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:44.364212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:44.364213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:44.364703 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:44.365006 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:44.365324 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 19:32:44.365562 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 19:32:44.366740 15263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:44.490514 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 15140
I20260923 19:32:44.510562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 14787
I20260923 19:32:44.511981 14850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:32:44.638666 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:44.638070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42795 (local address 127.0.0.1:43852)
0923 19:32:44.638281 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:44.638440 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 19:32:44.639000 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 14787
I20260923 19:32:44.661541 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--webserver_interface=127.0.97.62
--webserver_port=39987
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:32:45.045344 15315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:45.045738 15315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:45.045861 15315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:45.053879 15315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:32:45.054035 15315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:45.054103 15315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:45.054157 15315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:32:45.054208 15315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:32:45.065508 15315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:45.067543 15315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:45.069178 15315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:45.083317 15320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:45.083211 15321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:45.083253 15323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:45.084273 15315 server_base.cc:1034] running on GCE node
I20260923 19:32:45.085304 15315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:45.087267 15315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:45.088505 15315 hybrid_clock.cc:648] HybridClock initialized: now 1790191965088430 us; error 62 us; skew 500 ppm
I20260923 19:32:45.088920 15315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:45 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:45.093020 15315 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:32:45.095570 15315 webserver.cc:460] Webserver started at http://127.0.97.62:39987/ using document root <none> and password file <none>
I20260923 19:32:45.096431 15315 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:45.096546 15315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:45.103695 15315 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260923 19:32:45.107731 15330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:45.109635 15315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260923 19:32:45.109877 15315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "8134fe0d94a64de6a110832539a4f405"
format_stamp: "Formatted at 2026-09-23 19:32:38 on dist-test-slave-kxfb"
server_key: "b97fba90af9299ed56090554b9b6476c"
server_key_iv: "799a1693aae4c8baffb3c9493763a605"
server_key_version: "encryptionkey@0"
I20260923 19:32:45.111172 15315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:45.130646 15315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:45.135613 15315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:45.135939 15315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:45.157351 15315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43013
I20260923 19:32:45.157397 15381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43013 every 8 connection(s)
I20260923 19:32:45.159307 15315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:32:45.166807 15382 sys_catalog.cc:263] Verifying existing consensus state
I20260923 19:32:45.169557 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 15315
I20260923 19:32:45.170517 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Bootstrap starting.
I20260923 19:32:45.170624 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42795
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43013
--webserver_port=37931
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:32:45.193341 15382 log.cc:824] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:45.210651 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 19:32:45.211248 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405: Bootstrap complete.
I20260923 19:32:45.220871 15382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:45.221918 15382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8134fe0d94a64de6a110832539a4f405, State: Initialized, Role: FOLLOWER
I20260923 19:32:45.222607 15382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:45.222855 15382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:32:45.222965 15382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:32:45.223130 15382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 2 FOLLOWER]: Advancing to term 3
I20260923 19:32:45.227298 15382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:45.227856 15382 leader_election.cc:302] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8134fe0d94a64de6a110832539a4f405; no voters:
I20260923 19:32:45.228523 15382 leader_election.cc:288] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 19:32:45.228664 15386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 19:32:45.230113 15386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [term 3 LEADER]: Becoming Leader. State: Replica: 8134fe0d94a64de6a110832539a4f405, State: Running, Role: LEADER
I20260923 19:32:45.230778 15386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } }
I20260923 19:32:45.231114 15382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:32:45.233840 15387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8134fe0d94a64de6a110832539a4f405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260923 19:32:45.234264 15387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:45.233821 15388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8134fe0d94a64de6a110832539a4f405. Latest consensus state: current_term: 3 leader_uuid: "8134fe0d94a64de6a110832539a4f405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8134fe0d94a64de6a110832539a4f405" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43013 } } }
I20260923 19:32:45.234889 15388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8134fe0d94a64de6a110832539a4f405 [sys.catalog]: This master's current role is: LEADER
I20260923 19:32:45.245831 15392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:32:45.257715 15392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:32:45.260445 15392 catalog_manager.cc:1259] Loaded cluster ID: 76b60dec10ff4e8488346fbbfe9f6e2e
I20260923 19:32:45.260699 15392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 19:32:45.263455 15392 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6c68dd79ad google::LogMessage::Fail() at ??:0
@ 0x7f6c68dda88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6c68dd74de google::LogMessage::Flush() at ??:0
@ 0x7f6c68dd8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6c7e80a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6c7e807976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6c7e88f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6c7e88f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6c6ba23345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6c6ba2691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6c6ba26799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6c6ba01eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6c6ecdb6db start_thread at ??:0
@ 0x7f6c67c7161f clone at ??:0
W20260923 19:32:45.590512 15384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:45.590896 15384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:45.591037 15384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:45.599109 15384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:45.599255 15384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:45.599424 15384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:32:45.610877 15384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37931
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:45.612583 15384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:45.614145 15384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:45.628556 15422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:45.628826 15423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:45.629158 15425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:45.629757 15384 server_base.cc:1034] running on GCE node
I20260923 19:32:45.630513 15384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:45.631773 15384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:45.632987 15384 hybrid_clock.cc:648] HybridClock initialized: now 1790191965632932 us; error 45 us; skew 500 ppm
I20260923 19:32:45.633333 15384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:45 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:45.636927 15384 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:32:45.638819 15384 webserver.cc:460] Webserver started at http://127.0.97.1:37931/ using document root <none> and password file <none>
I20260923 19:32:45.639619 15384 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:45.639739 15384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:45.645710 15384 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 19:32:45.649235 15432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:45.651044 15384 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260923 19:32:45.651233 15384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "8f1c8e4712f641288dde8ce6427a8a7e"
format_stamp: "Formatted at 2026-09-23 19:32:39 on dist-test-slave-kxfb"
server_key: "d75a6a2ccec94d36cd0a3e50e37ccd32"
server_key_iv: "29bc918032cfdfae9bef4021a6450beb"
server_key_version: "encryptionkey@0"
I20260923 19:32:45.652261 15384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:45.669868 15384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:45.674240 15384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:45.674526 15384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:45.675634 15384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:45.677721 15439 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260923 19:32:45.684887 15384 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 19:32:45.685039 15384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260923 19:32:45.685174 15384 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 19:32:45.688638 15384 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 19:32:45.688723 15384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260923 19:32:45.689208 15439 tablet_bootstrap.cc:492] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Bootstrap starting.
I20260923 19:32:45.699894 15439 log.cc:824] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Log is configured to *not* fsync() on all Append() calls
I20260923 19:32:45.718636 15439 tablet_bootstrap.cc:492] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 19:32:45.719434 15439 tablet_bootstrap.cc:492] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Bootstrap complete.
I20260923 19:32:45.719935 15439 ts_tablet_manager.cc:1397] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.000s
I20260923 19:32:45.729337 15439 raft_consensus.cc:348] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:45.730623 15439 raft_consensus.cc:729] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1c8e4712f641288dde8ce6427a8a7e, State: Initialized, Role: FOLLOWER
I20260923 19:32:45.731547 15439 consensus_queue.cc:260] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:45.733384 15439 ts_tablet_manager.cc:1428] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260923 19:32:45.738312 15384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42795
I20260923 19:32:45.738423 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42795 every 8 connection(s)
I20260923 19:32:45.739739 15384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:32:45.742059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 15384
I20260923 19:32:45.742746 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44133
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43013
--webserver_port=37731
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:32:45.745824 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:45.743145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:39177)
0923 19:32:45.744265 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:45.744530 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":422,"threads_started":1}
W20260923 19:32:45.746837 15548 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43013 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43013: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
I20260923 19:32:45.747975 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:45.747518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:52315)
0923 19:32:45.747721 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:45.747836 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 19:32:45.749120 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:45.748728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:57879)
0923 19:32:45.748885 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:45.748977 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 19:32:45.749400 15548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 19:32:46.118685 15551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:46.119122 15551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:46.119261 15551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:46.127326 15551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:46.127485 15551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:46.127657 15551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:32:46.138693 15551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37731
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:46.140412 15551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:46.141997 15551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:46.157970 15556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:46.157889 15559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:46.157948 15557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:46.158869 15551 server_base.cc:1034] running on GCE node
I20260923 19:32:46.159629 15551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:46.160856 15551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:46.162096 15551 hybrid_clock.cc:648] HybridClock initialized: now 1790191966162023 us; error 60 us; skew 500 ppm
I20260923 19:32:46.162578 15551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:46 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:46.166344 15551 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:32:46.168179 15551 webserver.cc:460] Webserver started at http://127.0.97.2:37731/ using document root <none> and password file <none>
I20260923 19:32:46.169030 15551 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:46.169159 15551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:46.175760 15551 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:32:46.179212 15566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:46.181031 15551 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:32:46.181209 15551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "c4b83ca6fbd643cb931810f78512ad66"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "caa2afe31adcd2e534e3cb4da8b104e7"
server_key_iv: "a6b86d66706b897343d589d388a2444c"
server_key_version: "encryptionkey@0"
I20260923 19:32:46.182400 15551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:46.194932 15551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:46.200448 15551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:46.200742 15551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:46.201900 15551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:46.203492 15551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:46.203588 15551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:46.203675 15551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:46.203732 15551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:46.238355 15551 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44133
I20260923 19:32:46.238471 15678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44133 every 8 connection(s)
I20260923 19:32:46.239928 15551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:32:46.241607 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 15551
I20260923 19:32:46.242159 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42093
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43013
--webserver_port=42311
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:32:46.246404 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:46.243632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:41895)
0923 19:32:46.244674 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:46.244969 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":377,"threads_started":1}
W20260923 19:32:46.247203 15679 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43013 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43013: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
I20260923 19:32:46.248412 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:46.247949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:59781)
0923 19:32:46.248148 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:46.248248 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 19:32:46.249439 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:46.249080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:54051)
0923 19:32:46.249220 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:46.249306 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 19:32:46.249819 15679 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 19:32:46.586561 15682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:32:46.586862 15682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:32:46.586967 15682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:32:46.594177 15682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:32:46.594326 15682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:32:46.594458 15682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:32:46.604297 15682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42311
--tserver_master_addrs=127.0.97.62:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:32:46.605530 15682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:32:46.606886 15682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:32:46.620143 15687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:46.620210 15688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:32:46.620143 15690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:32:46.621551 15682 server_base.cc:1034] running on GCE node
I20260923 19:32:46.622254 15682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:32:46.623390 15682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:32:46.624604 15682 hybrid_clock.cc:648] HybridClock initialized: now 1790191966624550 us; error 50 us; skew 500 ppm
I20260923 19:32:46.624933 15682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:32:46 dist-test-slave-kxfb krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:32:46.627997 15682 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:32:46.629515 15682 webserver.cc:460] Webserver started at http://127.0.97.3:42311/ using document root <none> and password file <none>
I20260923 19:32:46.630179 15682 fs_manager.cc:359] Metadata directory not provided
I20260923 19:32:46.630309 15682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:32:46.635911 15682 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:32:46.638940 15697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:46.640441 15682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 19:32:46.640595 15682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "17532166823145ab87ea861bbc005d3f"
format_stamp: "Formatted at 2026-09-23 19:32:40 on dist-test-slave-kxfb"
server_key: "cfa0b0bb54a1e445316cdc6e1f27de13"
server_key_iv: "bc93b09e212e680de78f0360326869e6"
server_key_version: "encryptionkey@0"
I20260923 19:32:46.641552 15682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:32:46.678448 15682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:32:46.682586 15682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:32:46.682842 15682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:32:46.683882 15682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:32:46.685259 15682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:32:46.685352 15682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:46.685467 15682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:32:46.685541 15682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:32:46.719386 15682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42093
I20260923 19:32:46.719508 15809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42093 every 8 connection(s)
I20260923 19:32:46.720886 15682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:32:46.726827 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:46.724461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:49767)
0923 19:32:46.725333 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:46.725642 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":321,"threads_started":1}
W20260923 19:32:46.727705 15810 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43013 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43013: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
I20260923 19:32:46.728339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 15682
I20260923 19:32:46.728893 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:46.728412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:56621)
0923 19:32:46.728628 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:46.728729 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 19:32:46.729866 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:46.729512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:41051)
0923 19:32:46.729655 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:46.729745 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 19:32:46.730168 15810 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 19:32:46.751008 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:46.750031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:41389)
0923 19:32:46.750643 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:46.750752 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":284,"threads_started":1}
I20260923 19:32:47.251753 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:47.250656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:48409)
0923 19:32:47.251281 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:47.251449 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":278,"threads_started":1}
I20260923 19:32:47.573963 15816 raft_consensus.cc:482] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:32:47.574381 15816 raft_consensus.cc:504] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:47.576020 15816 leader_election.cc:288] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093), c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133)
Sep 23 19:32:47 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 23 19:32:47 dist-test-slave-kxfb krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:32:47.603001 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:47.576687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47463 (local address 127.0.97.3:42093)
0923 19:32:47.579382 (+ 2695us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:47.579400 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:47.579449 (+ 49us) server_negotiation.cc:407] Connection header received
0923 19:32:47.581856 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:47.581872 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:47.582555 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:47.582985 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:47.586145 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:47.587545 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:47.589701 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:47.590423 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:47.590587 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:47.595191 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:47.595199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:47.595215 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:47.595261 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:47.598270 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:47.599379 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:47.599388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:47.599398 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:47.599636 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:47.600146 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:47.600150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:47.600152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:47.600863 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:47.601003 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:47.602098 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0923 19:32:47.602574 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2459,"thread_start_us":2299,"threads_started":1}
I20260923 19:32:47.603037 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:47.576480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42093 (local address 127.0.97.1:47463)
0923 19:32:47.578895 (+ 2415us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:47.578920 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:47.579046 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:47.583296 (+ 4250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:47.583312 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:47.584530 (+ 1218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:47.585780 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:47.585814 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:47.587811 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:47.587817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:47.589531 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:47.589549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:47.590697 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:47.590704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:47.591003 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:47.591963 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:47.591998 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:47.594867 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:47.598507 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:47.598520 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:47.598564 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:47.599220 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:47.599769 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:47.599773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:47.599775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:47.600028 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:47.601029 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:47.601038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:47.601447 (+ 409us) client_negotiation.cc:769] Sending connection context
0923 19:32:47.601809 (+ 362us) client_negotiation.cc:240] Negotiation successful
0923 19:32:47.602494 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2280,"mutex_wait_us":328,"spinlock_wait_cycles":12160,"thread_start_us":259,"threads_started":1}
I20260923 19:32:47.603132 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:47.576480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44133 (local address 127.0.97.1:38523)
0923 19:32:47.577149 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:47.577208 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 19:32:47.578059 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:32:47.583296 (+ 5237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:32:47.583312 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:32:47.584530 (+ 1218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:32:47.585780 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:47.585814 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:47.587814 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:47.587819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:47.589537 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:32:47.589554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:32:47.590728 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:32:47.590735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:32:47.591004 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:47.591784 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:32:47.591817 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:32:47.594501 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:32:47.599381 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:47.599386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:47.599391 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:47.599932 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:47.600440 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:32:47.600445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:32:47.600449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:32:47.600609 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:32:47.601616 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:32:47.601619 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:32:47.601804 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 19:32:47.601979 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 19:32:47.602694 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":551,"spinlock_wait_cycles":26112,"thread_start_us":2115,"threads_started":1}
I20260923 19:32:47.603214 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:47.576825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38523 (local address 127.0.97.2:44133)
0923 19:32:47.579680 (+ 2855us) server_negotiation.cc:206] Beginning negotiation
0923 19:32:47.579693 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:32:47.579736 (+ 43us) server_negotiation.cc:407] Connection header received
0923 19:32:47.581972 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:32:47.581982 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:32:47.582548 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:32:47.582985 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:32:47.586075 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:47.587606 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:47.589707 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:32:47.590444 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:32:47.590640 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:32:47.594906 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:32:47.594919 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:32:47.594945 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:32:47.595012 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:32:47.599171 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:47.600089 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:47.600100 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:47.600108 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:47.600299 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:32:47.600738 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:32:47.600741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:32:47.600743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:32:47.601450 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:32:47.601664 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 19:32:47.602275 (+ 611us) server_negotiation.cc:299] Negotiation successful
0923 19:32:47.602823 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2640,"thread_start_us":2445,"threads_started":1}
W20260923 19:32:47.606771 15436 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
W20260923 19:32:47.607121 15433 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
I20260923 19:32:47.607281 15433 leader_election.cc:302] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1c8e4712f641288dde8ce6427a8a7e; no voters: 17532166823145ab87ea861bbc005d3f, c4b83ca6fbd643cb931810f78512ad66
I20260923 19:32:47.607757 15816 raft_consensus.cc:2726] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 19:32:47.731992 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:47.730931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:54169)
0923 19:32:47.731537 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:47.731698 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":258,"threads_started":1}
I20260923 19:32:47.752774 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:47.752215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:32779)
0923 19:32:47.752453 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:47.752603 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 19:32:48.253844 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:48.252942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:49453)
0923 19:32:48.253545 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:48.253646 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":336,"threads_started":1}
I20260923 19:32:48.734063 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:48.733174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:47959)
0923 19:32:48.733714 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:48.733814 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":291,"threads_started":1}
I20260923 19:32:48.754882 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:48.753875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:52181)
0923 19:32:48.754538 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:48.754654 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":306,"threads_started":1}
I20260923 19:32:49.183132 15825 raft_consensus.cc:482] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:32:49.183415 15825 raft_consensus.cc:504] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:49.184186 15825 leader_election.cc:288] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093), c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133)
W20260923 19:32:49.185286 15433 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
W20260923 19:32:49.185446 15436 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
I20260923 19:32:49.185572 15436 leader_election.cc:302] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1c8e4712f641288dde8ce6427a8a7e; no voters: 17532166823145ab87ea861bbc005d3f, c4b83ca6fbd643cb931810f78512ad66
I20260923 19:32:49.185835 15825 raft_consensus.cc:2726] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 19:32:49.255889 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:49.254935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:34451)
0923 19:32:49.255583 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:49.255680 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":346,"threads_started":1}
I20260923 19:32:49.735888 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:49.735079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:44383)
0923 19:32:49.735608 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:49.735693 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":255,"threads_started":1}
I20260923 19:32:49.757031 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:49.755977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:58519)
0923 19:32:49.756629 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:49.756765 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":333,"threads_started":1}
I20260923 19:32:50.257881 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:50.257060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:35199)
0923 19:32:50.257561 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:50.257653 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":261,"threads_started":1}
I20260923 19:32:50.697958 15830 raft_consensus.cc:482] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:32:50.698240 15830 raft_consensus.cc:504] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:50.699083 15830 leader_election.cc:288] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093), c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133)
W20260923 19:32:50.700189 15433 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
W20260923 19:32:50.700371 15436 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
I20260923 19:32:50.700515 15436 leader_election.cc:302] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1c8e4712f641288dde8ce6427a8a7e; no voters: 17532166823145ab87ea861bbc005d3f, c4b83ca6fbd643cb931810f78512ad66
I20260923 19:32:50.700783 15830 raft_consensus.cc:2726] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 19:32:50.738184 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:50.737223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:44353)
0923 19:32:50.737880 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:50.737965 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":383,"threads_started":1}
I20260923 19:32:50.759368 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:50.758396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:41239)
0923 19:32:50.758997 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:50.759092 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":270,"threads_started":1}
I20260923 19:32:51.259984 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:51.259041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:41713)
0923 19:32:51.259585 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:51.259713 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":292,"threads_started":1}
I20260923 19:32:51.740715 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:51.739391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:34985)
0923 19:32:51.740380 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:51.740496 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":305,"threads_started":1}
I20260923 19:32:51.761222 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:51.760388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:55623)
0923 19:32:51.760922 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:51.761015 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":276,"threads_started":1}
I20260923 19:32:52.262080 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:52.261141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:44297)
0923 19:32:52.261711 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:52.261847 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":271,"threads_started":1}
I20260923 19:32:52.742502 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:52.741695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:60467)
0923 19:32:52.742201 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:52.742282 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":292,"threads_started":1}
I20260923 19:32:52.763187 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:52.762224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:42731)
0923 19:32:52.762776 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:52.762870 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":324,"threads_started":1}
I20260923 19:32:53.264020 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:53.263038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:33267)
0923 19:32:53.263667 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:53.263766 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":282,"threads_started":1}
I20260923 19:32:53.739595 15840 raft_consensus.cc:482] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:32:53.739841 15840 raft_consensus.cc:504] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:32:53.740705 15840 leader_election.cc:288] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093), c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133)
W20260923 19:32:53.741956 15433 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
W20260923 19:32:53.742190 15436 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
I20260923 19:32:53.742374 15436 leader_election.cc:302] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1c8e4712f641288dde8ce6427a8a7e; no voters: 17532166823145ab87ea861bbc005d3f, c4b83ca6fbd643cb931810f78512ad66
I20260923 19:32:53.742691 15840 raft_consensus.cc:2726] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 19:32:53.744252 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:53.743357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:42687)
0923 19:32:53.743951 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:53.744050 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":303,"threads_started":1}
I20260923 19:32:53.765192 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:53.764219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:47417)
0923 19:32:53.764869 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:53.764965 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":366,"threads_started":1}
I20260923 19:32:54.266021 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:54.265200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:54913)
0923 19:32:54.265718 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:54.265813 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":290,"threads_started":1}
I20260923 19:32:54.745913 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:54.745135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:48371)
0923 19:32:54.745605 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:54.745701 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260923 19:32:54.767004 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:54.766114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:54183)
0923 19:32:54.766687 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:54.766773 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":349,"threads_started":1}
I20260923 19:32:55.268066 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:55.267130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:46851)
0923 19:32:55.267750 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:55.267862 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":339,"threads_started":1}
I20260923 19:32:55.747812 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:55.746938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:37819)
0923 19:32:55.747490 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:55.747617 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":260,"threads_started":1}
I20260923 19:32:55.769057 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:55.768194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:54543)
0923 19:32:55.768753 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:55.768853 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":331,"threads_started":1}
I20260923 19:32:56.270313 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:56.269327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:41559)
0923 19:32:56.269964 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:56.270068 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":327,"threads_started":1}
I20260923 19:32:56.749861 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:56.748924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:51489)
0923 19:32:56.749483 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:56.749627 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":290,"threads_started":1}
I20260923 19:32:56.771260 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:56.770139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:45955)
0923 19:32:56.770805 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:56.770970 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":388,"threads_started":1}
I20260923 19:32:57.272461 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:57.271473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:41115)
0923 19:32:57.272100 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:57.272208 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":305,"threads_started":1}
I20260923 19:32:57.751844 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:57.750996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:56683)
0923 19:32:57.751543 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:57.751643 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":296,"threads_started":1}
I20260923 19:32:57.773283 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:57.772384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:49539)
0923 19:32:57.772960 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:57.773068 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":341,"threads_started":1}
I20260923 19:32:58.274638 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:58.273709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:34959)
0923 19:32:58.274323 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:58.274414 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":277,"threads_started":1}
I20260923 19:32:58.753657 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:58.752813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:51141)
0923 19:32:58.753354 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:58.753439 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":280,"threads_started":1}
I20260923 19:32:58.775269 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:58.774406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:46553)
0923 19:32:58.774925 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:58.775038 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":288,"threads_started":1}
I20260923 19:32:59.276458 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:59.275595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:40315)
0923 19:32:59.276148 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:59.276242 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":266,"threads_started":1}
I20260923 19:32:59.755522 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:59.754673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:55743)
0923 19:32:59.755188 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:59.755287 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":254,"threads_started":1}
I20260923 19:32:59.777130 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:32:59.776256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:33783)
0923 19:32:59.776801 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 19:32:59.776902 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":292,"threads_started":1}
I20260923 19:33:00.029412 15861 raft_consensus.cc:482] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:33:00.029721 15861 raft_consensus.cc:504] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17532166823145ab87ea861bbc005d3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42093 } } peers { permanent_uuid: "c4b83ca6fbd643cb931810f78512ad66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44133 } } peers { permanent_uuid: "8f1c8e4712f641288dde8ce6427a8a7e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42795 } }
I20260923 19:33:00.030594 15861 leader_election.cc:288] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093), c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133)
W20260923 19:33:00.031693 15436 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17532166823145ab87ea861bbc005d3f (127.0.97.3:42093): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
W20260923 19:33:00.031858 15433 leader_election.cc:341] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4b83ca6fbd643cb931810f78512ad66 (127.0.97.2:44133): Not found: Tablet not found: fa8de6a4ac4a4d06908d9c423e0f9d38
I20260923 19:33:00.031980 15433 leader_election.cc:302] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f1c8e4712f641288dde8ce6427a8a7e; no voters: 17532166823145ab87ea861bbc005d3f, c4b83ca6fbd643cb931810f78512ad66
I20260923 19:33:00.032269 15861 raft_consensus.cc:2726] T fa8de6a4ac4a4d06908d9c423e0f9d38 P 8f1c8e4712f641288dde8ce6427a8a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 19:33:00.278591 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:00.277689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:47567)
0923 19:33:00.278207 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:00.278347 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":264,"threads_started":1}
I20260923 19:33:00.757686 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:00.756768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:55119)
0923 19:33:00.757305 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:00.757418 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":275,"threads_started":1}
I20260923 19:33:00.779832 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:00.778684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:46553)
0923 19:33:00.779467 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:00.779575 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":519,"threads_started":1}
I20260923 19:33:01.280689 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:01.279879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.2:50911)
0923 19:33:01.280361 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:01.280456 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":250,"threads_started":1}
I20260923 19:33:01.735249 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 15384
I20260923 19:33:01.756747 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:01.736446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.0.97.1:42795)
0923 19:33:01.737117 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:01.737129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:01.737163 (+ 34us) server_negotiation.cc:407] Connection header received
0923 19:33:01.737257 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:01.737268 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:01.737627 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:01.737828 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:01.739664 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:01.741345 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:01.742715 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:01.743677 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:01.743805 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:01.750858 (+ 7053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:01.750873 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:01.750892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:01.750936 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:01.753936 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:01.754674 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:01.754683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:01.754685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:01.754821 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:01.755104 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:01.755108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:01.755110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:01.755517 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:01.755805 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:01.756151 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 19:33:01.756394 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":327,"threads_started":1}
I20260923 19:33:01.758345 15543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:01.762513 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:01.759081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.3:34565)
0923 19:33:01.762189 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:01.762315 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2985,"thread_start_us":2837,"threads_started":1}
I20260923 19:33:01.852964 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:01.851820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43013 (local address 127.0.97.1:40377)
0923 19:33:01.852462 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:01.852595 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":320,"threads_started":1}
I20260923 19:33:01.852957 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 15384
I20260923 19:33:01.871232 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 15551
I20260923 19:33:01.884689 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:01.872053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.0.97.2:44133)
0923 19:33:01.872521 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:01.872526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:01.872544 (+ 18us) server_negotiation.cc:407] Connection header received
0923 19:33:01.872608 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:01.872612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:01.872706 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:01.872806 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:01.874154 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:01.875051 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:01.876262 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:01.876691 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:01.876779 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:01.879601 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:01.879606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:01.879610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:01.879665 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:01.882331 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:01.882964 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:01.882968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:01.882970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:01.883103 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:01.883387 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:01.883389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:01.883391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:01.883669 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:01.883881 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:01.884106 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 19:33:01.884281 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":242,"threads_started":1}
I20260923 19:33:01.885416 15674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:01.976096 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 15551
I20260923 19:33:01.993888 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 15682
I20260923 19:33:02.008878 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:01.994873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.0.97.3:42093)
0923 19:33:01.995466 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:01.995472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:01.995493 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:33:01.995562 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:01.995568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:01.995644 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:01.995746 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:01.997253 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:01.998142 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:01.999680 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:02.000141 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:02.000239 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:02.003222 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:02.003227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:02.003231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:02.003273 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:02.006185 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:02.007009 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:02.007013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:02.007015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:02.007180 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:02.007512 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:02.007515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:02.007517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:02.007839 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:02.008070 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:02.008315 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 19:33:02.008503 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":316,"threads_started":1}
I20260923 19:33:02.009734 15805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:02.097958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 15682
2026-09-23T19:33:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24806 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:33:02 dist-test-slave-kxfb krb5kdc[15876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:33:02 dist-test-slave-kxfb krb5kdc[15876](info): set up 2 sockets
Sep 23 19:33:02 dist-test-slave-kxfb krb5kdc[15876](info): commencing operation
krb5kdc: starting...
W20260923 19:33:03.155025 388 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.017s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:33:03 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:33:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:33:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:33:03.427481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38589
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:33:03.804800 15889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:03.805153 15889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:03.805277 15889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:03.812902 15889 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 19:33:03.813019 15889 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 19:33:03.813058 15889 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 19:33:03.813124 15889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:33:03.813198 15889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:03.813293 15889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:03.813351 15889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:33:03.813406 15889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:33:03.824162 15889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:03.825824 15889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:03.827478 15889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:03.840250 15894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:03.840167 15895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:03.840209 15897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:03.841327 15889 server_base.cc:1034] running on GCE node
I20260923 19:33:03.842180 15889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:03.843822 15889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:03.845043 15889 hybrid_clock.cc:648] HybridClock initialized: now 1790191983844961 us; error 57 us; skew 500 ppm
I20260923 19:33:03.845396 15889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:03 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:03.848738 15889 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:33:03.850704 15889 webserver.cc:460] Webserver started at http://127.0.97.62:35389/ using document root <none> and password file <none>
I20260923 19:33:03.851476 15889 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:03.851563 15889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:03.851852 15889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:33:03.855175 15889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "e8d0f9b54bb54487991a3a8a6ce86e92"
format_stamp: "Formatted at 2026-09-23 19:33:03 on dist-test-slave-kxfb"
server_key: "0be2b5855f9094ad11e79f56fc254c31"
server_key_iv: "54ba1b0ab20ccced0fd86f06b55437f4"
server_key_version: "encryptionkey@0"
I20260923 19:33:03.855974 15889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "e8d0f9b54bb54487991a3a8a6ce86e92"
format_stamp: "Formatted at 2026-09-23 19:33:03 on dist-test-slave-kxfb"
server_key: "0be2b5855f9094ad11e79f56fc254c31"
server_key_iv: "54ba1b0ab20ccced0fd86f06b55437f4"
server_key_version: "encryptionkey@0"
I20260923 19:33:03.861619 15889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:33:03.865324 15904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:03.866822 15889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 19:33:03.866971 15889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "e8d0f9b54bb54487991a3a8a6ce86e92"
format_stamp: "Formatted at 2026-09-23 19:33:03 on dist-test-slave-kxfb"
server_key: "0be2b5855f9094ad11e79f56fc254c31"
server_key_iv: "54ba1b0ab20ccced0fd86f06b55437f4"
server_key_version: "encryptionkey@0"
I20260923 19:33:03.867156 15889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:03.891762 15889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:03.896808 15889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:03.897078 15889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:03.916595 15889 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38589
I20260923 19:33:03.916620 15955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38589 every 8 connection(s)
I20260923 19:33:03.918152 15889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:33:03.922508 15956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:33:03.926019 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 15889
I20260923 19:33:03.926280 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:33:03.927182 388 external_mini_cluster.cc:1428] Setting key 21c89faf75babe873bcdb57cd60f661b
I20260923 19:33:03.936854 15956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Bootstrap starting.
I20260923 19:33:03.940585 15956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Neither blocks nor log segments found. Creating new log.
I20260923 19:33:03.941706 15956 log.cc:824] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Log is configured to *not* fsync() on all Append() calls
Sep 23 19:33:03 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:33:03.945279 15956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: No bootstrap required, opened a new log
I20260923 19:33:03.953050 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:03.929821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.0.97.62:38589)
0923 19:33:03.931093 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:03.931106 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:03.931153 (+ 47us) server_negotiation.cc:407] Connection header received
0923 19:33:03.932988 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:03.933015 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:03.933512 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:03.933963 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:03.935607 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:03.937049 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:03.938454 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:03.939206 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:03.939379 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:03.944075 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:03.944087 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:03.944109 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:03.944158 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:03.948281 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:03.949125 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:03.949134 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:03.949144 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:03.949372 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:03.949858 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:03.949863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:03.949867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:03.950594 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:03.950854 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:03.951339 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 19:33:03.951824 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":759,"threads_started":1}
I20260923 19:33:03.955353 15956 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:03.955853 15956 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:33:03.955967 15956 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d0f9b54bb54487991a3a8a6ce86e92, State: Initialized, Role: FOLLOWER
I20260923 19:33:03.956648 15956 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:03.956887 15956 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:33:03.956985 15956 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:33:03.957122 15956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:33:03.959542 15956 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:03.960073 15956 leader_election.cc:302] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8d0f9b54bb54487991a3a8a6ce86e92; no voters:
I20260923 19:33:03.960662 15956 leader_election.cc:288] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:33:03.960819 15961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:33:03.961894 15961 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 1 LEADER]: Becoming Leader. State: Replica: e8d0f9b54bb54487991a3a8a6ce86e92, State: Running, Role: LEADER
I20260923 19:33:03.962512 15961 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:03.963169 15956 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:33:03.965451 15962 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } } }
I20260923 19:33:03.965879 15962 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: This master's current role is: LEADER
I20260923 19:33:03.966744 15963 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8d0f9b54bb54487991a3a8a6ce86e92. Latest consensus state: current_term: 1 leader_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } } }
I20260923 19:33:03.967249 15963 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: This master's current role is: LEADER
I20260923 19:33:03.968482 15967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:33:03.973711 15967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:33:03.983510 15967 catalog_manager.cc:1347] Generated new cluster ID: 94dffc4be6f943229659164657bea261
I20260923 19:33:03.983662 15967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:33:04.007923 15967 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:33:04.009500 15967 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:33:04.018323 15967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Generated new TSK 0
I20260923 19:33:04.019424 15967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:33:04.094044 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38589
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:33:04.470692 15982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:04.471002 15982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:04.471132 15982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:04.478484 15982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:04.478595 15982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:04.478727 15982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:33:04.488503 15982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:04.489821 15982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:04.491132 15982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:04.504694 15987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:04.504810 15990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:04.504796 15988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:04.505805 15982 server_base.cc:1034] running on GCE node
I20260923 19:33:04.506516 15982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:04.507469 15982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:04.508701 15982 hybrid_clock.cc:648] HybridClock initialized: now 1790191984508625 us; error 64 us; skew 500 ppm
I20260923 19:33:04.509038 15982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:04 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:04.512624 15982 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:33:04.514406 15982 webserver.cc:460] Webserver started at http://127.0.97.1:38513/ using document root <none> and password file <none>
I20260923 19:33:04.515136 15982 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:04.515259 15982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:04.515569 15982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:33:04.518610 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "761512a07d6349c286a19261d3e33f56"
format_stamp: "Formatted at 2026-09-23 19:33:04 on dist-test-slave-kxfb"
server_key: "ded2b107b70fe8a0767c7571644f0392"
server_key_iv: "604c8114e81e9a5c1cd4e6c24c761bdb"
server_key_version: "encryptionkey@0"
I20260923 19:33:04.519395 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "761512a07d6349c286a19261d3e33f56"
format_stamp: "Formatted at 2026-09-23 19:33:04 on dist-test-slave-kxfb"
server_key: "ded2b107b70fe8a0767c7571644f0392"
server_key_iv: "604c8114e81e9a5c1cd4e6c24c761bdb"
server_key_version: "encryptionkey@0"
I20260923 19:33:04.525106 15982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 19:33:04.529029 15997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:04.530656 15982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 19:33:04.530835 15982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "761512a07d6349c286a19261d3e33f56"
format_stamp: "Formatted at 2026-09-23 19:33:04 on dist-test-slave-kxfb"
server_key: "ded2b107b70fe8a0767c7571644f0392"
server_key_iv: "604c8114e81e9a5c1cd4e6c24c761bdb"
server_key_version: "encryptionkey@0"
I20260923 19:33:04.531023 15982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:04.553871 15982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:04.558935 15982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:04.559269 15982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:04.560456 15982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:33:04.562083 15982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:33:04.562177 15982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:04.562336 15982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:33:04.562394 15982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 19:33:04.597373 15982 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41667
I20260923 19:33:04.597422 16109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41667 every 8 connection(s)
I20260923 19:33:04.598960 15982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:33:04.607375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 15982
I20260923 19:33:04.607596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:33:04.608332 388 external_mini_cluster.cc:1428] Setting key f4f89b2d9d25c28a5c565f5b4e6529b8
Sep 23 19:33:04 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:33:04.621516 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:04.602133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35875 (local address 127.0.97.62:38589)
0923 19:33:04.602724 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:04.602729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:04.603823 (+ 1094us) server_negotiation.cc:407] Connection header received
0923 19:33:04.605808 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:04.605814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:04.605918 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:04.606023 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:04.607752 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:04.608542 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:04.610133 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:04.610753 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:04.610902 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:04.615542 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:04.615548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:04.615552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:04.615600 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:04.618245 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:04.619104 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:04.619108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:04.619111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:04.619234 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:04.619666 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:04.619669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:04.619671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:04.619965 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:04.620109 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:04.620857 (+ 748us) server_negotiation.cc:299] Negotiation successful
0923 19:33:04.621104 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":285,"threads_started":1}
I20260923 19:33:04.622628 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:04.602792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:35875)
0923 19:33:04.603624 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:04.603670 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:04.605500 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:04.606329 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:04.606342 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:04.606908 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:33:04.607527 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:04.607559 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:04.608694 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:04.608698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:04.609969 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:04.609984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:04.611004 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:04.611011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:04.611264 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:04.612153 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:33:04.612178 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:33:04.615194 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:33:04.618444 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:04.618455 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:04.618475 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:04.618971 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:04.619341 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:04.619346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:04.619350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:04.619547 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:04.620091 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:33:04.620100 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:33:04.620452 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 19:33:04.620707 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 19:33:04.621269 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":311,"threads_started":1}
I20260923 19:33:04.624563 16110 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38589
I20260923 19:33:04.624970 16110 heartbeater.cc:464] Registering TS with master...
I20260923 19:33:04.625846 16110 heartbeater.cc:511] Master 127.0.97.62:38589 requested a full tablet report, sending...
I20260923 19:33:04.628259 15921 ts_manager.cc:194] Registered new tserver with Master: 761512a07d6349c286a19261d3e33f56 (127.0.97.1:41667)
I20260923 19:33:04.629909 15921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35875
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:33:04.664201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38589
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:33:05.036602 16116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:05.036981 16116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:05.037092 16116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:05.045300 16116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:05.045454 16116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:05.045630 16116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:33:05.056716 16116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:05.058441 16116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:05.060070 16116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:05.075142 16122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:05.075037 16121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:05.075683 16124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:05.076884 16116 server_base.cc:1034] running on GCE node
I20260923 19:33:05.077665 16116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:05.078918 16116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:05.080150 16116 hybrid_clock.cc:648] HybridClock initialized: now 1790191985080084 us; error 52 us; skew 500 ppm
I20260923 19:33:05.080497 16116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:05 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:05.083952 16116 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:33:05.085672 16116 webserver.cc:460] Webserver started at http://127.0.97.2:44165/ using document root <none> and password file <none>
I20260923 19:33:05.086511 16116 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:05.086632 16116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:05.086934 16116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:33:05.090205 16116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "7934e8b7fdeaab0822df49125ed9866e"
server_key_iv: "a64699d5d1b568481f550c4da131a679"
server_key_version: "encryptionkey@0"
I20260923 19:33:05.091033 16116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "7934e8b7fdeaab0822df49125ed9866e"
server_key_iv: "a64699d5d1b568481f550c4da131a679"
server_key_version: "encryptionkey@0"
I20260923 19:33:05.096990 16116 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 19:33:05.101328 16131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:05.103067 16116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:33:05.103237 16116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "7934e8b7fdeaab0822df49125ed9866e"
server_key_iv: "a64699d5d1b568481f550c4da131a679"
server_key_version: "encryptionkey@0"
I20260923 19:33:05.103442 16116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:05.131294 16116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:05.136216 16116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:05.136502 16116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:05.137650 16116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:33:05.139335 16116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:33:05.139424 16116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:05.139518 16116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:33:05.139565 16116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:05.174786 16116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39801
I20260923 19:33:05.174831 16243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39801 every 8 connection(s)
I20260923 19:33:05.176281 16116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:33:05.181687 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 16116
I20260923 19:33:05.181872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:33:05.182382 388 external_mini_cluster.cc:1428] Setting key 531ec29dd7c0812208f5633874f3ac44
Sep 23 19:33:05 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:33:05.199455 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.179212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36239 (local address 127.0.97.62:38589)
0923 19:33:05.179706 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:05.179712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:05.180642 (+ 930us) server_negotiation.cc:407] Connection header received
0923 19:33:05.182745 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:05.182751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:05.182855 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:05.182970 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:05.184886 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.185709 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.188073 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.188547 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.188701 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.193162 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:05.193167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:05.193171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:05.193207 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:05.196056 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:05.196839 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:05.196843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:05.196845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:05.196988 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:05.197428 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:05.197431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:05.197433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:05.197752 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:05.197909 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:05.198769 (+ 860us) server_negotiation.cc:299] Negotiation successful
0923 19:33:05.199038 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":250,"threads_started":1}
I20260923 19:33:05.200703 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.179687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:36239)
0923 19:33:05.180434 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:05.180480 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:05.182454 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:05.183238 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:05.183251 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:05.183907 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:33:05.184698 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.184725 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.185870 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.185876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.187907 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.187929 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.188832 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.188839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.189132 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.189941 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:33:05.189973 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:33:05.192886 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:33:05.196214 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:05.196223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:05.196240 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:05.196707 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:05.197097 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:05.197100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:05.197103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:05.197308 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:05.197879 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:33:05.197888 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:33:05.198345 (+ 457us) client_negotiation.cc:769] Sending connection context
0923 19:33:05.198632 (+ 287us) client_negotiation.cc:240] Negotiation successful
0923 19:33:05.199304 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":291,"threads_started":1}
I20260923 19:33:05.202497 16244 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38589
I20260923 19:33:05.202883 16244 heartbeater.cc:464] Registering TS with master...
I20260923 19:33:05.203719 16244 heartbeater.cc:511] Master 127.0.97.62:38589 requested a full tablet report, sending...
I20260923 19:33:05.205508 15921 ts_manager.cc:194] Registered new tserver with Master: 03d1f4dcbc1e4244baf8f1e4d98bb3c8 (127.0.97.2:39801)
I20260923 19:33:05.206513 15921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36239
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:33:05.245289 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38589
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:33:05.614831 16250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:05.615170 16250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:05.615278 16250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:05.622963 16250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:05.623095 16250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:05.623229 16250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:33:05.633531 16110 heartbeater.cc:503] Master 127.0.97.62:38589 was elected leader, sending a full tablet report...
I20260923 19:33:05.633944 16250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:05.635919 16250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:05.637501 16250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:05.652189 16256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:05.652264 16255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:05.652245 16258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:05.652865 16250 server_base.cc:1034] running on GCE node
I20260923 19:33:05.653677 16250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:05.654825 16250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:05.656061 16250 hybrid_clock.cc:648] HybridClock initialized: now 1790191985655999 us; error 51 us; skew 500 ppm
I20260923 19:33:05.656409 16250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:05 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:05.659840 16250 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:33:05.661514 16250 webserver.cc:460] Webserver started at http://127.0.97.3:34577/ using document root <none> and password file <none>
I20260923 19:33:05.662340 16250 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:05.662453 16250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:05.662762 16250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:33:05.666184 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "4799dd750b3e46d2b46df4ab8e655115"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "bca995437577b8777882bc8bcd645115"
server_key_iv: "b5a2a74b6ed5f61580ef50cf1c158b16"
server_key_version: "encryptionkey@0"
I20260923 19:33:05.667007 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4799dd750b3e46d2b46df4ab8e655115"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "bca995437577b8777882bc8bcd645115"
server_key_iv: "b5a2a74b6ed5f61580ef50cf1c158b16"
server_key_version: "encryptionkey@0"
I20260923 19:33:05.672829 16250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:33:05.676909 16265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:05.678639 16250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:33:05.678850 16250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "4799dd750b3e46d2b46df4ab8e655115"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "bca995437577b8777882bc8bcd645115"
server_key_iv: "b5a2a74b6ed5f61580ef50cf1c158b16"
server_key_version: "encryptionkey@0"
I20260923 19:33:05.679044 16250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:05.696772 16250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:05.701249 16250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:05.701516 16250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:05.702564 16250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:33:05.703977 16250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:33:05.704049 16250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:05.704154 16250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:33:05.704200 16250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:05.737541 16250 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39237
I20260923 19:33:05.737577 16377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39237 every 8 connection(s)
I20260923 19:33:05.739141 16250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:33:05.742924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 16250
I20260923 19:33:05.743115 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:33:05.743609 388 external_mini_cluster.cc:1428] Setting key 9683bf695f5d925d52a896a1e74e7b3f
Sep 23 19:33:05 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:33:05.761159 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.742127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54741 (local address 127.0.97.62:38589)
0923 19:33:05.742776 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:05.742784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:05.744039 (+ 1255us) server_negotiation.cc:407] Connection header received
0923 19:33:05.746083 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:05.746088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:05.746195 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:05.746349 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:05.748023 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.748874 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.750531 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.750940 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.751032 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.754831 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:05.754836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:05.754840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:05.754875 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:05.757688 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:05.758734 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:05.758738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:05.758741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:05.758903 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:05.759412 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:05.759417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:05.759420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:05.759738 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:05.759861 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:05.760658 (+ 797us) server_negotiation.cc:299] Negotiation successful
0923 19:33:05.760861 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":245,"threads_started":1}
I20260923 19:33:05.762341 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.742873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:54741)
0923 19:33:05.743848 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:05.743900 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:05.745818 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:05.746609 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:05.746619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:05.747211 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:33:05.747868 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.747886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.749020 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.749024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.750422 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.750435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.751120 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.751125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.751370 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.752017 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:33:05.752041 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:33:05.754609 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:33:05.757836 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:05.757846 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:05.757863 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:05.758594 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:05.759009 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:05.759013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:05.759015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:05.759286 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:05.759872 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:33:05.759879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:33:05.760299 (+ 420us) client_negotiation.cc:769] Sending connection context
0923 19:33:05.760563 (+ 264us) client_negotiation.cc:240] Negotiation successful
0923 19:33:05.761032 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":368,"threads_started":1}
I20260923 19:33:05.763815 16378 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38589
I20260923 19:33:05.764132 16378 heartbeater.cc:464] Registering TS with master...
I20260923 19:33:05.764917 16378 heartbeater.cc:511] Master 127.0.97.62:38589 requested a full tablet report, sending...
I20260923 19:33:05.766513 15921 ts_manager.cc:194] Registered new tserver with Master: 4799dd750b3e46d2b46df4ab8e655115 (127.0.97.3:39237)
I20260923 19:33:05.767521 15921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54741
I20260923 19:33:05.771883 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:33:05.789903 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.776744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.0.97.62:38589)
0923 19:33:05.776922 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:05.776927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:05.777435 (+ 508us) server_negotiation.cc:407] Connection header received
0923 19:33:05.777579 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:05.777622 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:05.777697 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:05.777782 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:05.779217 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.780022 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.781331 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.781752 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.781851 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.784574 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:05.784581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:05.784585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:05.784631 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:05.787324 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:05.787976 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:05.787980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:05.787983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:05.788137 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:05.788439 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:05.788442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:05.788444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:05.788875 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:05.789092 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:05.789351 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 19:33:05.789618 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 19:33:05.794862 15921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:33:05.802987 15921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:33:05.840844 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.817410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:48634)
0923 19:33:05.831180 (+ 13770us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:05.831200 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:05.831335 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:05.832289 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:05.832295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:05.832328 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:05.832866 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.832878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.835164 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.835171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.837567 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.837586 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.839811 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.839818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.840009 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.840049 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 19:33:05.840194 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 19:33:05.840420 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13644,"thread_start_us":12350,"threads_started":1}
I20260923 19:33:05.841543 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.818060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.0.97.1:41667)
0923 19:33:05.829670 (+ 11610us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:05.829683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:05.831322 (+ 1639us) server_negotiation.cc:407] Connection header received
0923 19:33:05.831489 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:05.831497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:05.831761 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:05.832003 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:05.833071 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.834960 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.837755 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.839306 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.839421 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.839814 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:05.841016 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0923 19:33:05.841199 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11392,"thread_start_us":11209,"threads_started":1}
I20260923 19:33:05.842077 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.817474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.0.97.3:39237)
0923 19:33:05.831222 (+ 13748us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:05.831232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:05.831271 (+ 39us) server_negotiation.cc:407] Connection header received
0923 19:33:05.831387 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:05.831397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:05.831734 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:05.832907 (+ 1173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:05.833242 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.835084 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.839262 (+ 4178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.840518 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.840653 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.840882 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:05.841543 (+ 661us) server_negotiation.cc:299] Negotiation successful
0923 19:33:05.841719 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13546,"thread_start_us":13351,"threads_started":1}
I20260923 19:33:05.842344 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.817113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39237 (local address 127.0.0.1:41696)
0923 19:33:05.830211 (+ 13098us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:05.830273 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:05.830551 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:05.832030 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:05.832040 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:05.832079 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:05.832641 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.832657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.835265 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.835272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.839063 (+ 3791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.839078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.840719 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.840726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.840907 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.841384 (+ 477us) client_negotiation.cc:769] Sending connection context
0923 19:33:05.841471 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 19:33:05.841807 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12800,"thread_start_us":13813,"threads_started":1}
I20260923 19:33:05.846124 16045 tablet_service.cc:1467] Processing CreateTablet for tablet 86b094bee4074be2b16994efd64bddeb (DEFAULT_TABLE table=test-table [id=ff3712be81db4fe9bd8cb1d8bd4661e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:33:05.847641 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.831720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.0.97.2:39801)
0923 19:33:05.835943 (+ 4223us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:05.835954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:05.838511 (+ 2557us) server_negotiation.cc:407] Connection header received
0923 19:33:05.838643 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:05.838652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:05.838980 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:05.839220 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:05.840183 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.841600 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.844543 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.845985 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.846142 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.847032 (+ 890us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:05.847118 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 19:33:05.847301 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4019,"thread_start_us":3857,"threads_started":1}
I20260923 19:33:05.848008 16045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b094bee4074be2b16994efd64bddeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:33:05.848007 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.831684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39801 (local address 127.0.0.1:43618)
0923 19:33:05.838190 (+ 6506us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:05.838210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:05.838374 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:05.839511 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:05.839516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:05.839542 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:05.839985 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.839997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.842495 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.842499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.844358 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.844372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.846344 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.846362 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.846809 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.846888 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 19:33:05.847020 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 19:33:05.847182 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6357,"thread_start_us":6178,"threads_started":1}
I20260923 19:33:05.847757 16313 tablet_service.cc:1467] Processing CreateTablet for tablet 86b094bee4074be2b16994efd64bddeb (DEFAULT_TABLE table=test-table [id=ff3712be81db4fe9bd8cb1d8bd4661e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:33:05.849187 16313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b094bee4074be2b16994efd64bddeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:33:05.852442 16179 tablet_service.cc:1467] Processing CreateTablet for tablet 86b094bee4074be2b16994efd64bddeb (DEFAULT_TABLE table=test-table [id=ff3712be81db4fe9bd8cb1d8bd4661e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:33:05.853974 16179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b094bee4074be2b16994efd64bddeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:33:05.861285 16397 tablet_bootstrap.cc:492] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: Bootstrap starting.
I20260923 19:33:05.863539 16398 tablet_bootstrap.cc:492] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: Bootstrap starting.
I20260923 19:33:05.867089 16398 tablet_bootstrap.cc:654] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: Neither blocks nor log segments found. Creating new log.
I20260923 19:33:05.867084 16397 tablet_bootstrap.cc:654] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: Neither blocks nor log segments found. Creating new log.
I20260923 19:33:05.868026 16399 tablet_bootstrap.cc:492] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Bootstrap starting.
I20260923 19:33:05.868510 16398 log.cc:824] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: Log is configured to *not* fsync() on all Append() calls
I20260923 19:33:05.869232 16397 log.cc:824] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: Log is configured to *not* fsync() on all Append() calls
I20260923 19:33:05.871590 16399 tablet_bootstrap.cc:654] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Neither blocks nor log segments found. Creating new log.
I20260923 19:33:05.872189 16397 tablet_bootstrap.cc:492] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: No bootstrap required, opened a new log
I20260923 19:33:05.872491 16397 ts_tablet_manager.cc:1397] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 19:33:05.872686 16399 log.cc:824] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Log is configured to *not* fsync() on all Append() calls
I20260923 19:33:05.873621 16398 tablet_bootstrap.cc:492] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: No bootstrap required, opened a new log
I20260923 19:33:05.873891 16398 ts_tablet_manager.cc:1397] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 19:33:05.876197 16399 tablet_bootstrap.cc:492] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: No bootstrap required, opened a new log
I20260923 19:33:05.876458 16399 ts_tablet_manager.cc:1397] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 19:33:05.880614 16397 raft_consensus.cc:348] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } }
I20260923 19:33:05.881029 16397 raft_consensus.cc:374] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:33:05.881140 16397 raft_consensus.cc:729] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4799dd750b3e46d2b46df4ab8e655115, State: Initialized, Role: FOLLOWER
I20260923 19:33:05.881781 16397 consensus_queue.cc:260] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } }
I20260923 19:33:05.884473 16398 raft_consensus.cc:348] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } }
I20260923 19:33:05.884968 16398 raft_consensus.cc:374] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:33:05.885077 16398 raft_consensus.cc:729] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761512a07d6349c286a19261d3e33f56, State: Initialized, Role: FOLLOWER
I20260923 19:33:05.885097 16378 heartbeater.cc:503] Master 127.0.97.62:38589 was elected leader, sending a full tablet report...
I20260923 19:33:05.885782 16398 consensus_queue.cc:260] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } }
I20260923 19:33:05.886834 16399 raft_consensus.cc:348] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } }
I20260923 19:33:05.887290 16399 raft_consensus.cc:374] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:33:05.887383 16399 raft_consensus.cc:729] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d1f4dcbc1e4244baf8f1e4d98bb3c8, State: Initialized, Role: FOLLOWER
I20260923 19:33:05.887746 16397 ts_tablet_manager.cc:1428] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260923 19:33:05.887771 16398 ts_tablet_manager.cc:1428] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260923 19:33:05.888082 16399 consensus_queue.cc:260] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } }
I20260923 19:33:05.889170 16244 heartbeater.cc:503] Master 127.0.97.62:38589 was elected leader, sending a full tablet report...
I20260923 19:33:05.889777 16399 ts_tablet_manager.cc:1428] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260923 19:33:05.915462 16405 raft_consensus.cc:482] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:33:05.915841 16405 raft_consensus.cc:504] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } }
I20260923 19:33:05.918412 16405 leader_election.cc:288] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4799dd750b3e46d2b46df4ab8e655115 (127.0.97.3:39237), 761512a07d6349c286a19261d3e33f56 (127.0.97.1:41667)
I20260923 19:33:05.927405 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.918885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55805 (local address 127.0.97.3:39237)
0923 19:33:05.919136 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:05.919144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:05.920306 (+ 1162us) server_negotiation.cc:407] Connection header received
0923 19:33:05.920692 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:05.920698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:05.920798 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:05.920891 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:05.921854 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.922904 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.925065 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.926278 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.926454 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.926589 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:05.926970 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 19:33:05.927127 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 19:33:05.927600 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.918659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39237 (local address 127.0.97.2:55805)
0923 19:33:05.920156 (+ 1497us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:05.920181 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:05.920463 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:05.921084 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:05.921094 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:05.921199 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:05.921682 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.921695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.923061 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.923066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.924916 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.924929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.926543 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.926550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.926770 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.926818 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 19:33:05.926933 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 19:33:05.927100 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"mutex_wait_us":1,"spinlock_wait_cycles":27904,"thread_start_us":1060,"threads_started":1}
I20260923 19:33:05.928042 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.919332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58061 (local address 127.0.97.1:41667)
0923 19:33:05.919551 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:05.919558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:05.920581 (+ 1023us) server_negotiation.cc:407] Connection header received
0923 19:33:05.920742 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:05.920747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:05.920810 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:05.920885 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:05.921468 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.922510 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.925305 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.926587 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.926709 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.926815 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:05.927159 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 19:33:05.927303 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 19:33:05.928376 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.918659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.97.2:58061)
0923 19:33:05.920468 (+ 1809us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:05.920482 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:05.920612 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:05.920923 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:05.920928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:05.920952 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:05.921303 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.921313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.922637 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.922644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.925134 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:05.925150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.926801 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.926807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:05.926977 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.927016 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 19:33:05.927807 (+ 791us) client_negotiation.cc:240] Negotiation successful
0923 19:33:05.927954 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1705,"mutex_wait_us":262,"thread_start_us":1297,"threads_started":1}
W20260923 19:33:05.928515 16245 tablet.cc:2369] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:33:05.929075 16333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b094bee4074be2b16994efd64bddeb" candidate_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4799dd750b3e46d2b46df4ab8e655115" is_pre_election: true
I20260923 19:33:05.929467 16065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b094bee4074be2b16994efd64bddeb" candidate_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761512a07d6349c286a19261d3e33f56" is_pre_election: true
I20260923 19:33:05.929885 16333 raft_consensus.cc:2454] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03d1f4dcbc1e4244baf8f1e4d98bb3c8 in term 0.
I20260923 19:33:05.930085 16065 raft_consensus.cc:2454] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03d1f4dcbc1e4244baf8f1e4d98bb3c8 in term 0.
I20260923 19:33:05.930657 16133 leader_election.cc:302] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d1f4dcbc1e4244baf8f1e4d98bb3c8, 4799dd750b3e46d2b46df4ab8e655115; no voters:
I20260923 19:33:05.931089 16405 raft_consensus.cc:2781] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:33:05.931253 16405 raft_consensus.cc:482] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:33:05.931367 16405 raft_consensus.cc:3037] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:33:05.935062 16405 raft_consensus.cc:504] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } }
I20260923 19:33:05.935948 16405 leader_election.cc:288] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4799dd750b3e46d2b46df4ab8e655115 (127.0.97.3:39237), 761512a07d6349c286a19261d3e33f56 (127.0.97.1:41667)
I20260923 19:33:05.937749 16065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b094bee4074be2b16994efd64bddeb" candidate_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761512a07d6349c286a19261d3e33f56"
I20260923 19:33:05.938004 16065 raft_consensus.cc:3037] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:33:05.941546 16065 raft_consensus.cc:2454] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03d1f4dcbc1e4244baf8f1e4d98bb3c8 in term 1.
I20260923 19:33:05.942173 16132 leader_election.cc:302] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d1f4dcbc1e4244baf8f1e4d98bb3c8, 761512a07d6349c286a19261d3e33f56; no voters:
I20260923 19:33:05.942731 16405 raft_consensus.cc:2781] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:33:05.943639 16333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b094bee4074be2b16994efd64bddeb" candidate_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4799dd750b3e46d2b46df4ab8e655115"
I20260923 19:33:05.943835 16405 raft_consensus.cc:686] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 1 LEADER]: Becoming Leader. State: Replica: 03d1f4dcbc1e4244baf8f1e4d98bb3c8, State: Running, Role: LEADER
I20260923 19:33:05.943931 16333 raft_consensus.cc:3037] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:33:05.944518 16405 consensus_queue.cc:237] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } }
I20260923 19:33:05.946918 16333 raft_consensus.cc:2454] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03d1f4dcbc1e4244baf8f1e4d98bb3c8 in term 1.
I20260923 19:33:05.950076 15921 catalog_manager.cc:5495] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03d1f4dcbc1e4244baf8f1e4d98bb3c8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } health_report { overall_health: HEALTHY } } }
I20260923 19:33:05.982883 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:05.977153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.0.97.2:39801)
0923 19:33:05.977389 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:05.977395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:05.977416 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:33:05.977539 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:05.977543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:05.977652 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:05.977791 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:33:05.978178 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.979011 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.980526 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:05.980954 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:05.981069 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:05.981575 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:33:05.982221 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:33:05.982424 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:05.982477 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 19:33:05.982587 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 19:33:05.990968 16379 tablet.cc:2369] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:33:05.992851 16333 raft_consensus.cc:1261] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [term 1 FOLLOWER]: Refusing update from remote peer 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:33:05.992851 16065 raft_consensus.cc:1261] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [term 1 FOLLOWER]: Refusing update from remote peer 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:33:05.994030 16405 consensus_queue.cc:1035] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:33:05.994429 16408 consensus_queue.cc:1035] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:33:06.020394 15920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:33:06.020722 15920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:33:06.025022 15920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Sending DeleteTablet for 3 replicas of tablet 86b094bee4074be2b16994efd64bddeb
I20260923 19:33:06.026453 16313 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b094bee4074be2b16994efd64bddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:33:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41696
I20260923 19:33:06.026541 16045 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b094bee4074be2b16994efd64bddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:33:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48634
I20260923 19:33:06.026685 16179 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b094bee4074be2b16994efd64bddeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:33:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43618
I20260923 19:33:06.028179 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 15982
I20260923 19:33:06.041967 16421 tablet_replica.cc:330] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: stopping tablet replica
I20260923 19:33:06.042687 16421 raft_consensus.cc:2229] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:33:06.043392 16421 raft_consensus.cc:2258] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:33:06.046663 16421 ts_tablet_manager.cc:1905] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:33:06.054016 16422 tablet_replica.cc:330] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: stopping tablet replica
I20260923 19:33:06.054682 16422 raft_consensus.cc:2229] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:33:06.055644 16422 raft_consensus.cc:2258] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:33:06.059595 16421 ts_tablet_manager.cc:1918] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:33:06.060353 16420 tablet_replica.cc:330] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: stopping tablet replica
I20260923 19:33:06.060817 16420 raft_consensus.cc:2229] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:33:06.061391 16420 raft_consensus.cc:2258] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:33:06.061902 16421 log.cc:1196] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/86b094bee4074be2b16994efd64bddeb
I20260923 19:33:06.062196 16422 ts_tablet_manager.cc:1905] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:33:06.062551 16421 ts_tablet_manager.cc:1939] T 86b094bee4074be2b16994efd64bddeb P 761512a07d6349c286a19261d3e33f56: Deleting consensus metadata
I20260923 19:33:06.064178 16420 ts_tablet_manager.cc:1905] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:33:06.065235 15905 catalog_manager.cc:4841] TS 761512a07d6349c286a19261d3e33f56 (127.0.97.1:41667): tablet 86b094bee4074be2b16994efd64bddeb (table test-table [id=ff3712be81db4fe9bd8cb1d8bd4661e8]) successfully deleted
Sep 23 19:33:06 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:33:06.072803 16420 ts_tablet_manager.cc:1918] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:33:06.072803 16422 ts_tablet_manager.cc:1918] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:33:06.073024 16422 log.cc:1196] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/86b094bee4074be2b16994efd64bddeb
I20260923 19:33:06.073287 16420 log.cc:1196] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/86b094bee4074be2b16994efd64bddeb
I20260923 19:33:06.073606 16422 ts_tablet_manager.cc:1939] T 86b094bee4074be2b16994efd64bddeb P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Deleting consensus metadata
I20260923 19:33:06.073947 16420 ts_tablet_manager.cc:1939] T 86b094bee4074be2b16994efd64bddeb P 4799dd750b3e46d2b46df4ab8e655115: Deleting consensus metadata
I20260923 19:33:06.076056 15906 catalog_manager.cc:4841] TS 4799dd750b3e46d2b46df4ab8e655115 (127.0.97.3:39237): tablet 86b094bee4074be2b16994efd64bddeb (table test-table [id=ff3712be81db4fe9bd8cb1d8bd4661e8]) successfully deleted
I20260923 19:33:06.076368 15906 catalog_manager.cc:4841] TS 03d1f4dcbc1e4244baf8f1e4d98bb3c8 (127.0.97.2:39801): tablet 86b094bee4074be2b16994efd64bddeb (table test-table [id=ff3712be81db4fe9bd8cb1d8bd4661e8]) successfully deleted
I20260923 19:33:06.080644 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:06.029662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.0.97.1:41667)
0923 19:33:06.029880 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:06.029888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:06.044732 (+ 14844us) server_negotiation.cc:407] Connection header received
0923 19:33:06.050785 (+ 6053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:06.050793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:06.050902 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:06.050994 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:06.056175 (+ 5181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:06.057510 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:06.063827 (+ 6317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:06.064619 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:06.064776 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:06.072236 (+ 7460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:06.072267 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:06.072293 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:06.072362 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:06.077005 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:06.077950 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:06.077962 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:06.077966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:06.078181 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:06.078650 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:06.078656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:06.078660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:06.079242 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:06.079602 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:06.079875 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 19:33:06.080146 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 19:33:06.081511 16105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:06.180150 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 15982
I20260923 19:33:06.205001 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 16116
Sep 23 19:33:06 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:33:06.224509 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:06.206040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.0.97.2:39801)
0923 19:33:06.206320 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:06.206327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:06.206347 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:33:06.206426 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:06.206430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:06.206525 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:06.206618 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:06.208261 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:06.209271 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:06.210976 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:06.211466 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:06.211591 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:06.216813 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:06.216834 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:06.216856 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:06.216933 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:06.221078 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:06.222006 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:06.222016 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:06.222020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:06.222197 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:06.222664 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:06.222669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:06.222672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:06.223172 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:06.223540 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:06.223842 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 19:33:06.224097 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 19:33:06.225610 16239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:06.352384 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 16116
I20260923 19:33:06.368816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 16250
Sep 23 19:33:06 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790191983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:33:06.385578 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:06.369607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.0.97.3:39237)
0923 19:33:06.369858 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:06.369863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:06.369884 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:33:06.369962 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:06.369967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:06.370050 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:06.370154 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:06.371620 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:06.372501 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:06.373882 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:06.374340 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:06.374442 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:06.379200 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:06.379222 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:06.379239 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:06.379299 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:06.382165 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:06.383274 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:06.383282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:06.383284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:06.383506 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:06.383959 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:06.383962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:06.383965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:06.384377 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:06.384649 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:06.385048 (+ 399us) server_negotiation.cc:299] Negotiation successful
0923 19:33:06.385264 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260923 19:33:06.386425 16373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:06.476912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 16250
I20260923 19:33:06.490839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 15889
I20260923 19:33:06.492138 15951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:06.607131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 15889
I20260923 19:33:26.626495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38589
--webserver_interface=127.0.97.62
--webserver_port=35389
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:33:26.979112 16428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:26.979405 16428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:26.979511 16428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:26.986788 16428 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 19:33:26.986884 16428 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 19:33:26.986925 16428 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 19:33:26.986979 16428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:33:26.987066 16428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:26.987167 16428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:26.987228 16428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:33:26.987280 16428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:33:26.997321 16428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:26.998678 16428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:27.000002 16428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:27.025780 16435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:27.025812 16437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:27.025748 16434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:27.026429 16428 server_base.cc:1034] running on GCE node
I20260923 19:33:27.027712 16428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:27.029757 16428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:27.031030 16428 hybrid_clock.cc:648] HybridClock initialized: now 1790192007030946 us; error 42 us; skew 500 ppm
I20260923 19:33:27.031530 16428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:27 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:27.035597 16428 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:33:27.037793 16428 webserver.cc:460] Webserver started at http://127.0.97.62:35389/ using document root <none> and password file <none>
I20260923 19:33:27.038842 16428 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:27.038977 16428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:27.046687 16428 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 19:33:27.050640 16444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:27.052459 16428 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:33:27.052670 16428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "e8d0f9b54bb54487991a3a8a6ce86e92"
format_stamp: "Formatted at 2026-09-23 19:33:03 on dist-test-slave-kxfb"
server_key: "0be2b5855f9094ad11e79f56fc254c31"
server_key_iv: "54ba1b0ab20ccced0fd86f06b55437f4"
server_key_version: "encryptionkey@0"
I20260923 19:33:27.053907 16428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:27.087132 16428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:27.092056 16428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:27.092384 16428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:27.113981 16428 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38589
I20260923 19:33:27.114022 16495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38589 every 8 connection(s)
I20260923 19:33:27.116014 16428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:33:27.123638 16496 sys_catalog.cc:263] Verifying existing consensus state
I20260923 19:33:27.125144 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 16428
I20260923 19:33:27.126025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41667
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38589
--webserver_port=38513
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:33:27.127221 16496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Bootstrap starting.
I20260923 19:33:27.146965 16496 log.cc:824] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Log is configured to *not* fsync() on all Append() calls
I20260923 19:33:27.159087 16496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 19:33:27.159651 16496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Bootstrap complete.
I20260923 19:33:27.168939 16496 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:27.169854 16496 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d0f9b54bb54487991a3a8a6ce86e92, State: Initialized, Role: FOLLOWER
I20260923 19:33:27.170529 16496 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:27.170773 16496 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:33:27.170887 16496 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:33:27.171038 16496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 1 FOLLOWER]: Advancing to term 2
I20260923 19:33:27.175657 16496 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:27.176260 16496 leader_election.cc:302] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8d0f9b54bb54487991a3a8a6ce86e92; no voters:
I20260923 19:33:27.177035 16496 leader_election.cc:288] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 19:33:27.177178 16500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 19:33:27.178401 16500 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 2 LEADER]: Becoming Leader. State: Replica: e8d0f9b54bb54487991a3a8a6ce86e92, State: Running, Role: LEADER
I20260923 19:33:27.179054 16500 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:27.179526 16496 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:33:27.181718 16501 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } } }
I20260923 19:33:27.182183 16501 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: This master's current role is: LEADER
I20260923 19:33:27.186461 16502 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8d0f9b54bb54487991a3a8a6ce86e92. Latest consensus state: current_term: 2 leader_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } } }
I20260923 19:33:27.186826 16502 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: This master's current role is: LEADER
I20260923 19:33:27.188738 16505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:33:27.198221 16505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:33:27.199859 16505 catalog_manager.cc:1259] Loaded cluster ID: 94dffc4be6f943229659164657bea261
I20260923 19:33:27.199972 16505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:33:27.204988 16505 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:33:27.206764 16505 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Loaded TSK: 0
I20260923 19:33:27.238235 16505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Generated new TSK 1
I20260923 19:33:27.238687 16505 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 19:33:27.539637 16498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:27.539987 16498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:27.540112 16498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:27.547983 16498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:27.548130 16498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:27.548285 16498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:33:27.559480 16498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38513
--tserver_master_addrs=127.0.97.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:27.561079 16498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:27.562708 16498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:27.577719 16524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:27.577687 16526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:27.577793 16523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:27.583917 16498 server_base.cc:1034] running on GCE node
I20260923 19:33:27.584662 16498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:27.585729 16498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:27.586947 16498 hybrid_clock.cc:648] HybridClock initialized: now 1790192007586886 us; error 54 us; skew 500 ppm
I20260923 19:33:27.587299 16498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:27 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:27.590538 16498 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:33:27.592129 16498 webserver.cc:460] Webserver started at http://127.0.97.1:38513/ using document root <none> and password file <none>
I20260923 19:33:27.592945 16498 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:27.593092 16498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:27.599010 16498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 19:33:27.602093 16533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:27.603751 16498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:33:27.603926 16498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "761512a07d6349c286a19261d3e33f56"
format_stamp: "Formatted at 2026-09-23 19:33:04 on dist-test-slave-kxfb"
server_key: "ded2b107b70fe8a0767c7571644f0392"
server_key_iv: "604c8114e81e9a5c1cd4e6c24c761bdb"
server_key_version: "encryptionkey@0"
I20260923 19:33:27.604974 16498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:27.621554 16498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:27.625793 16498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:27.626050 16498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:27.627132 16498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:33:27.628531 16498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:33:27.628623 16498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:27.628707 16498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:33:27.628770 16498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:27.661054 16498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41667
I20260923 19:33:27.661160 16645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41667 every 8 connection(s)
I20260923 19:33:27.662746 16498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:33:27.664654 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 16498
I20260923 19:33:27.665501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39801
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38589
--webserver_port=44165
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:33:27 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:33:27.695240 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:27.666662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:33841)
0923 19:33:27.667684 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:27.667743 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:27.670172 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:27.671910 (+ 1738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:27.671924 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:27.672785 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:33:27.673656 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:27.673684 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:27.675396 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:27.675402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:27.677066 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:27.677082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:27.678220 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:27.678225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:27.678533 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:27.679392 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:33:27.679423 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:33:27.682252 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:33:27.690237 (+ 7985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:27.690251 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:27.690273 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:27.690922 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:27.691510 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:27.691513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:27.691516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:27.691750 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:27.692749 (+ 999us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:33:27.692756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:33:27.693150 (+ 394us) client_negotiation.cc:769] Sending connection context
0923 19:33:27.693408 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 19:33:27.693926 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":377,"threads_started":1}
I20260923 19:33:27.695721 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:27.666472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33841 (local address 127.0.97.62:38589)
0923 19:33:27.668576 (+ 2104us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:27.668590 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:27.668635 (+ 45us) server_negotiation.cc:407] Connection header received
0923 19:33:27.670870 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:27.670885 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:27.671288 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:27.671626 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:27.673911 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:27.675215 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:27.677234 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:27.678010 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:27.678155 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:27.682668 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:27.682680 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:27.682704 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:27.682762 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:27.689961 (+ 7199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:27.691100 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:27.691110 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:27.691120 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:27.691384 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:27.691891 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:27.691896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:27.691900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:27.692600 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:27.692819 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:27.693737 (+ 918us) server_negotiation.cc:299] Negotiation successful
0923 19:33:27.694280 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1894,"thread_start_us":1765,"threads_started":1}
I20260923 19:33:27.699186 16646 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38589
I20260923 19:33:27.699589 16646 heartbeater.cc:464] Registering TS with master...
I20260923 19:33:27.700518 16646 heartbeater.cc:511] Master 127.0.97.62:38589 requested a full tablet report, sending...
I20260923 19:33:27.703184 16461 ts_manager.cc:194] Registered new tserver with Master: 761512a07d6349c286a19261d3e33f56 (127.0.97.1:41667)
I20260923 19:33:27.705353 16461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33841
W20260923 19:33:28.015996 16650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:28.016299 16650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:28.016419 16650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:28.025265 16650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:28.025396 16650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:28.025555 16650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:33:28.035589 16650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44165
--tserver_master_addrs=127.0.97.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:28.036861 16650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:28.038159 16650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:28.051419 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:28.051540 16655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:28.051508 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:28.052531 16650 server_base.cc:1034] running on GCE node
I20260923 19:33:28.053292 16650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:28.054520 16650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:28.055764 16650 hybrid_clock.cc:648] HybridClock initialized: now 1790192008055703 us; error 57 us; skew 500 ppm
I20260923 19:33:28.056128 16650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:28 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:28.059834 16650 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:33:28.061563 16650 webserver.cc:460] Webserver started at http://127.0.97.2:44165/ using document root <none> and password file <none>
I20260923 19:33:28.062439 16650 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:28.062569 16650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:28.069016 16650 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:33:28.072945 16665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:28.074863 16650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:33:28.075052 16650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "7934e8b7fdeaab0822df49125ed9866e"
server_key_iv: "a64699d5d1b568481f550c4da131a679"
server_key_version: "encryptionkey@0"
I20260923 19:33:28.076218 16650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:28.094830 16650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:28.099556 16650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:28.099828 16650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:28.100919 16650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:33:28.102409 16650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:33:28.102536 16650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:28.102660 16650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:33:28.102747 16650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:28.138875 16650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39801
I20260923 19:33:28.138994 16777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39801 every 8 connection(s)
I20260923 19:33:28.140604 16650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:33:28.150214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 16650
I20260923 19:33:28.150928 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39237
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38589
--webserver_port=34577
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 19:33:28 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:33:28.160406 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.143481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51137 (local address 127.0.97.62:38589)
0923 19:33:28.143653 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.143657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.144983 (+ 1326us) server_negotiation.cc:407] Connection header received
0923 19:33:28.146978 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.146983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.147093 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.147180 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:28.148711 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.149416 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.150973 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.151430 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.151536 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.154849 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:28.154855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:28.154858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:28.154891 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:28.157342 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:28.158123 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:28.158127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:28.158130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:28.158251 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:28.158677 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:28.158680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:28.158682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:28.159054 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:28.159193 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:28.159853 (+ 660us) server_negotiation.cc:299] Negotiation successful
0923 19:33:28.160079 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 19:33:28.161326 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.144000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:51137)
0923 19:33:28.144798 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:28.144851 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:28.146705 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:28.147448 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:28.147457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:28.147982 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:33:28.148552 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.148568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.149563 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.149567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.150840 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.150854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.151642 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.151649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.151869 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.152490 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:33:28.152512 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:33:28.154625 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:33:28.157468 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:28.157477 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:28.157493 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:28.158011 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:28.158368 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:28.158373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:28.158377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:28.158558 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:28.159170 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:33:28.159178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:33:28.159501 (+ 323us) client_negotiation.cc:769] Sending connection context
0923 19:33:28.159732 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 19:33:28.160226 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":307,"threads_started":1}
I20260923 19:33:28.162755 16778 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38589
I20260923 19:33:28.163234 16778 heartbeater.cc:464] Registering TS with master...
I20260923 19:33:28.164287 16778 heartbeater.cc:511] Master 127.0.97.62:38589 requested a full tablet report, sending...
I20260923 19:33:28.166065 16461 ts_manager.cc:194] Registered new tserver with Master: 03d1f4dcbc1e4244baf8f1e4d98bb3c8 (127.0.97.2:39801)
I20260923 19:33:28.166920 16461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51137
W20260923 19:33:28.534132 16781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:28.534508 16781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:28.534647 16781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:28.542511 16781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:28.542647 16781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:28.542806 16781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:33:28.553653 16781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34577
--tserver_master_addrs=127.0.97.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:28.555346 16781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:28.556849 16781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:28.571762 16786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:28.572373 16789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:28.572373 16787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:28.573439 16781 server_base.cc:1034] running on GCE node
I20260923 19:33:28.574162 16781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:28.575321 16781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:28.576537 16781 hybrid_clock.cc:648] HybridClock initialized: now 1790192008576484 us; error 46 us; skew 500 ppm
I20260923 19:33:28.576861 16781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:28 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:28.580099 16781 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:33:28.581900 16781 webserver.cc:460] Webserver started at http://127.0.97.3:34577/ using document root <none> and password file <none>
I20260923 19:33:28.582827 16781 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:28.582959 16781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:28.589372 16781 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:33:28.592635 16796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:28.594274 16781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 19:33:28.594482 16781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "4799dd750b3e46d2b46df4ab8e655115"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "bca995437577b8777882bc8bcd645115"
server_key_iv: "b5a2a74b6ed5f61580ef50cf1c158b16"
server_key_version: "encryptionkey@0"
I20260923 19:33:28.595495 16781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:28.622442 16781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:28.626621 16781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:28.626868 16781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:28.627923 16781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:33:28.629307 16781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:33:28.629406 16781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:28.629490 16781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:33:28.629546 16781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:28.673687 16781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39237
I20260923 19:33:28.673835 16908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39237 every 8 connection(s)
I20260923 19:33:28.675820 16781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:33:28.684019 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 16781
Sep 23 19:33:28 dist-test-slave-kxfb krb5kdc[15876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:33:28.700302 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.679525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36637 (local address 127.0.97.62:38589)
0923 19:33:28.680178 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.680184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.681549 (+ 1365us) server_negotiation.cc:407] Connection header received
0923 19:33:28.683596 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.683603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.683725 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.683870 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:28.685570 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.686643 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.688824 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.689413 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.689528 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.693379 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:28.693384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:28.693387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:28.693418 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:28.696340 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:28.697105 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:28.697111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:28.697115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:28.697266 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:28.697693 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:28.697699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:28.697702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:28.697994 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:28.698133 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:28.699600 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0923 19:33:28.699860 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":505,"thread_start_us":404,"threads_started":1}
I20260923 19:33:28.701030 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.685023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.0.97.62:38589)
0923 19:33:28.685525 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.685531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.686161 (+ 630us) server_negotiation.cc:407] Connection header received
0923 19:33:28.686369 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.686375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.686463 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.686601 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:28.688488 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.689597 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.691208 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.691632 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.691747 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.694757 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:28.694762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:28.694765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:28.694813 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:28.697604 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:28.698386 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:28.698389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:28.698392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:28.698510 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:28.698892 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:28.698896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:28.698899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:28.699218 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:28.699468 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:28.700417 (+ 949us) server_negotiation.cc:299] Negotiation successful
0923 19:33:28.700671 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260923 19:33:28.701108 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.680239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:36637)
0923 19:33:28.681305 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:28.681378 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:28.683265 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:28.684085 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:28.684094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:28.684635 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:33:28.685337 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.685361 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.686889 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.686894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.688682 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.688694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.689636 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.689644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.689869 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.690544 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:33:28.690565 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:33:28.693099 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:33:28.696469 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:28.696477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:28.696494 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:28.696980 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:28.697373 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:33:28.697379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:33:28.697383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:33:28.697582 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:33:28.698126 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:33:28.698135 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:33:28.698569 (+ 434us) client_negotiation.cc:769] Sending connection context
0923 19:33:28.698878 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 19:33:28.699582 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":391,"threads_started":1}
I20260923 19:33:28.702550 16909 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38589
I20260923 19:33:28.702942 16909 heartbeater.cc:464] Registering TS with master...
I20260923 19:33:28.703884 16909 heartbeater.cc:511] Master 127.0.97.62:38589 requested a full tablet report, sending...
I20260923 19:33:28.705364 16461 ts_manager.cc:194] Registered new tserver with Master: 4799dd750b3e46d2b46df4ab8e655115 (127.0.97.3:39237)
I20260923 19:33:28.706252 16461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36637
I20260923 19:33:28.709584 16646 heartbeater.cc:503] Master 127.0.97.62:38589 was elected leader, sending a full tablet report...
I20260923 19:33:28.716562 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:33:28.723507 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.717726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.0.97.62:38589)
0923 19:33:28.717941 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.717946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.718402 (+ 456us) server_negotiation.cc:407] Connection header received
0923 19:33:28.718602 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.718606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.718749 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.718898 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:33:28.719276 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.720017 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.721560 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.722003 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.722097 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.722691 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:33:28.722962 (+ 271us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 19:33:28.723270 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39144: token expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 19:33:28.723631 16914 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39144: token expired
W20260923 19:33:28.724310 16382 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:38589)
I20260923 19:33:28.737732 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.724834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.97.62:38589)
0923 19:33:28.724973 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.724977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.724994 (+ 17us) server_negotiation.cc:407] Connection header received
0923 19:33:28.725101 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.725104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.725166 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.725253 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:28.726838 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.727626 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.729094 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.729505 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.729605 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.732459 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:28.732464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:28.732467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:28.732507 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:28.735078 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:28.735853 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:28.735857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:28.735860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:28.736015 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:28.736342 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:28.736345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:28.736346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:28.736738 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:28.736930 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:28.737255 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 19:33:28.737461 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 19:33:28.741544 16461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:33:28.749010 16461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:33:28.788285 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.763686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:55730)
0923 19:33:28.777064 (+ 13378us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:28.777091 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:28.777272 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:28.778810 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:28.778819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:28.778852 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:28.780423 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.780444 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.783041 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.783046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.785369 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.785381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.787102 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.787108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.787333 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.787394 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 19:33:28.787573 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 19:33:28.787826 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13198,"thread_start_us":11525,"threads_started":1}
I20260923 19:33:28.789088 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.763332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.0.97.2:39801)
0923 19:33:28.779642 (+ 16310us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.779653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.779690 (+ 37us) server_negotiation.cc:407] Connection header received
0923 19:33:28.779814 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.779823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.780139 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.780378 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:28.781951 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.783510 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.786265 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.787659 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.787796 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.787970 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:28.788564 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 19:33:28.788735 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16114,"thread_start_us":17247,"threads_started":1}
I20260923 19:33:28.789245 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.764125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.0.97.1:41667)
0923 19:33:28.777925 (+ 13800us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.777935 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.777970 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:33:28.778090 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.778099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.778495 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.780707 (+ 2212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:28.781026 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.782889 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.785573 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.786880 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.787017 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.787276 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:28.788660 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0923 19:33:28.788892 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13600,"thread_start_us":13430,"threads_started":1}
I20260923 19:33:28.789835 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.763179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39801 (local address 127.0.0.1:35446)
0923 19:33:28.775795 (+ 12616us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:28.775867 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:28.776124 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:28.781134 (+ 5010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:28.781140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:28.781177 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:28.781644 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.781658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.783709 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.783715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.786053 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.786070 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.788144 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.788150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.788341 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.788375 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:33:28.789348 (+ 973us) client_negotiation.cc:240] Negotiation successful
0923 19:33:28.789524 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12224,"mutex_wait_us":1,"thread_start_us":13496,"threads_started":1}
I20260923 19:33:28.791103 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.778027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39237 (local address 127.0.0.1:36158)
0923 19:33:28.781719 (+ 3692us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:28.781739 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:28.781998 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:28.782572 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:28.782576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:28.782599 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:28.782925 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.782935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.784780 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.784785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.786736 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.786748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.790277 (+ 3529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.790311 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.790495 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.790529 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:33:28.790620 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 19:33:28.790770 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3555,"thread_start_us":3375,"threads_started":1}
I20260923 19:33:28.791244 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.777690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.0.97.3:39237)
0923 19:33:28.781178 (+ 3488us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.781186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.781837 (+ 651us) server_negotiation.cc:407] Connection header received
0923 19:33:28.782176 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.782184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.782408 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.782571 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:28.783104 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.784632 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.786923 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.788857 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.788984 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.789255 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:28.790675 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0923 19:33:28.790868 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3335,"thread_start_us":3167,"threads_started":1}
I20260923 19:33:28.793172 16581 tablet_service.cc:1467] Processing CreateTablet for tablet 144ae7878bf54028b87057067c251d7f (DEFAULT_TABLE table=test-table [id=2b7a26c8014a4ecf815013033f6768cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:33:28.793915 16713 tablet_service.cc:1467] Processing CreateTablet for tablet 144ae7878bf54028b87057067c251d7f (DEFAULT_TABLE table=test-table [id=2b7a26c8014a4ecf815013033f6768cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:33:28.794811 16581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144ae7878bf54028b87057067c251d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:33:28.797012 16844 tablet_service.cc:1467] Processing CreateTablet for tablet 144ae7878bf54028b87057067c251d7f (DEFAULT_TABLE table=test-table [id=2b7a26c8014a4ecf815013033f6768cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:33:28.798367 16844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144ae7878bf54028b87057067c251d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:33:28.798549 16713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144ae7878bf54028b87057067c251d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:33:28.810235 16926 tablet_bootstrap.cc:492] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Bootstrap starting.
I20260923 19:33:28.812980 16927 tablet_bootstrap.cc:492] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: Bootstrap starting.
I20260923 19:33:28.814723 16926 tablet_bootstrap.cc:654] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Neither blocks nor log segments found. Creating new log.
I20260923 19:33:28.814774 16928 tablet_bootstrap.cc:492] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Bootstrap starting.
I20260923 19:33:28.816490 16926 log.cc:824] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Log is configured to *not* fsync() on all Append() calls
I20260923 19:33:28.816968 16927 tablet_bootstrap.cc:654] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: Neither blocks nor log segments found. Creating new log.
I20260923 19:33:28.818303 16927 log.cc:824] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: Log is configured to *not* fsync() on all Append() calls
I20260923 19:33:28.818594 16928 tablet_bootstrap.cc:654] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Neither blocks nor log segments found. Creating new log.
I20260923 19:33:28.819840 16928 log.cc:824] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Log is configured to *not* fsync() on all Append() calls
I20260923 19:33:28.820971 16926 tablet_bootstrap.cc:492] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: No bootstrap required, opened a new log
I20260923 19:33:28.821369 16926 ts_tablet_manager.cc:1397] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 19:33:28.821556 16927 tablet_bootstrap.cc:492] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: No bootstrap required, opened a new log
I20260923 19:33:28.821831 16927 ts_tablet_manager.cc:1397] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 19:33:28.822671 16928 tablet_bootstrap.cc:492] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: No bootstrap required, opened a new log
I20260923 19:33:28.822924 16928 ts_tablet_manager.cc:1397] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 19:33:28.830991 16928 raft_consensus.cc:348] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } }
I20260923 19:33:28.831457 16928 raft_consensus.cc:374] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:33:28.831578 16928 raft_consensus.cc:729] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d1f4dcbc1e4244baf8f1e4d98bb3c8, State: Initialized, Role: FOLLOWER
I20260923 19:33:28.832266 16928 consensus_queue.cc:260] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } }
I20260923 19:33:28.832544 16927 raft_consensus.cc:348] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } }
I20260923 19:33:28.832544 16926 raft_consensus.cc:348] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } }
I20260923 19:33:28.833046 16926 raft_consensus.cc:374] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:33:28.833105 16927 raft_consensus.cc:374] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:33:28.833204 16927 raft_consensus.cc:729] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4799dd750b3e46d2b46df4ab8e655115, State: Initialized, Role: FOLLOWER
I20260923 19:33:28.833269 16926 raft_consensus.cc:729] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761512a07d6349c286a19261d3e33f56, State: Initialized, Role: FOLLOWER
I20260923 19:33:28.833577 16778 heartbeater.cc:503] Master 127.0.97.62:38589 was elected leader, sending a full tablet report...
I20260923 19:33:28.833962 16926 consensus_queue.cc:260] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } }
I20260923 19:33:28.833962 16927 consensus_queue.cc:260] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } }
I20260923 19:33:28.835346 16928 ts_tablet_manager.cc:1428] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260923 19:33:28.837033 16909 heartbeater.cc:503] Master 127.0.97.62:38589 was elected leader, sending a full tablet report...
I20260923 19:33:28.837750 16927 ts_tablet_manager.cc:1428] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260923 19:33:28.838830 16926 ts_tablet_manager.cc:1428] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
W20260923 19:33:28.892894 16779 tablet.cc:2369] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:33:28.916704 16647 tablet.cc:2369] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:33:28.927963 16910 tablet.cc:2369] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:33:28.962702 16934 raft_consensus.cc:482] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:33:28.963027 16934 raft_consensus.cc:504] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } }
I20260923 19:33:28.964818 16934 leader_election.cc:288] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d1f4dcbc1e4244baf8f1e4d98bb3c8 (127.0.97.2:39801), 761512a07d6349c286a19261d3e33f56 (127.0.97.1:41667)
I20260923 19:33:28.970768 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.965382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60665 (local address 127.0.97.1:41667)
0923 19:33:28.965587 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.965592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.965614 (+ 22us) server_negotiation.cc:407] Connection header received
0923 19:33:28.965874 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.965879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.965958 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.966042 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:28.966630 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.967393 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.969014 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.969856 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.969976 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.970091 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:28.970366 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 19:33:28.970492 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 19:33:28.970767 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.965172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.97.3:60665)
0923 19:33:28.965529 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:28.965552 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:28.965688 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:28.966250 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:28.966254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:28.966276 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:28.966484 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.966495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.967521 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.967525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.968859 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.968873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.970011 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.970017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.970192 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.970219 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 19:33:28.970339 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 19:33:28.970487 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260923 19:33:28.972136 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.965334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58827 (local address 127.0.97.2:39801)
0923 19:33:28.965591 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:28.965599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:28.966371 (+ 772us) server_negotiation.cc:407] Connection header received
0923 19:33:28.966607 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:28.966612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:28.966681 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:28.966767 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:33:28.967325 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.968084 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.970080 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.971203 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.971326 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.971434 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:28.971775 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 19:33:28.971939 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 19:33:28.972215 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:28.965287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39801 (local address 127.0.97.3:58827)
0923 19:33:28.966231 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:28.966254 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:33:28.966460 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:33:28.966792 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:33:28.966797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:33:28.966819 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:33:28.967178 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.967187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.968230 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.968235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.969868 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:33:28.969883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:33:28.971399 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:33:28.971405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:33:28.971567 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:28.971600 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 19:33:28.971702 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 19:33:28.971846 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":504,"threads_started":1}
I20260923 19:33:28.972266 16601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144ae7878bf54028b87057067c251d7f" candidate_uuid: "4799dd750b3e46d2b46df4ab8e655115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761512a07d6349c286a19261d3e33f56" is_pre_election: true
I20260923 19:33:28.972962 16601 raft_consensus.cc:2454] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4799dd750b3e46d2b46df4ab8e655115 in term 0.
I20260923 19:33:28.973340 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144ae7878bf54028b87057067c251d7f" candidate_uuid: "4799dd750b3e46d2b46df4ab8e655115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" is_pre_election: true
I20260923 19:33:28.973855 16733 raft_consensus.cc:2454] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4799dd750b3e46d2b46df4ab8e655115 in term 0.
I20260923 19:33:28.973912 16797 leader_election.cc:302] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4799dd750b3e46d2b46df4ab8e655115, 761512a07d6349c286a19261d3e33f56; no voters:
I20260923 19:33:28.974429 16934 raft_consensus.cc:2781] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:33:28.974612 16934 raft_consensus.cc:482] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:33:28.974725 16934 raft_consensus.cc:3037] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:33:28.977187 16934 raft_consensus.cc:504] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } }
I20260923 19:33:28.977919 16934 leader_election.cc:288] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [CANDIDATE]: Term 1 election: Requested vote from peers 03d1f4dcbc1e4244baf8f1e4d98bb3c8 (127.0.97.2:39801), 761512a07d6349c286a19261d3e33f56 (127.0.97.1:41667)
I20260923 19:33:28.978488 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144ae7878bf54028b87057067c251d7f" candidate_uuid: "4799dd750b3e46d2b46df4ab8e655115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8"
I20260923 19:33:28.978706 16733 raft_consensus.cc:3037] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:33:28.978644 16601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144ae7878bf54028b87057067c251d7f" candidate_uuid: "4799dd750b3e46d2b46df4ab8e655115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761512a07d6349c286a19261d3e33f56"
I20260923 19:33:28.978911 16601 raft_consensus.cc:3037] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:33:28.981412 16601 raft_consensus.cc:2454] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4799dd750b3e46d2b46df4ab8e655115 in term 1.
I20260923 19:33:28.981433 16733 raft_consensus.cc:2454] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4799dd750b3e46d2b46df4ab8e655115 in term 1.
I20260923 19:33:28.981913 16798 leader_election.cc:302] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d1f4dcbc1e4244baf8f1e4d98bb3c8, 4799dd750b3e46d2b46df4ab8e655115; no voters:
I20260923 19:33:28.982355 16934 raft_consensus.cc:2781] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:33:28.982903 16934 raft_consensus.cc:686] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 1 LEADER]: Becoming Leader. State: Replica: 4799dd750b3e46d2b46df4ab8e655115, State: Running, Role: LEADER
I20260923 19:33:28.983454 16934 consensus_queue.cc:237] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } }
I20260923 19:33:28.988777 16460 catalog_manager.cc:5495] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4799dd750b3e46d2b46df4ab8e655115 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4799dd750b3e46d2b46df4ab8e655115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4799dd750b3e46d2b46df4ab8e655115" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39237 } health_report { overall_health: HEALTHY } } }
I20260923 19:33:29.031798 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.025394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.0.97.3:39237)
0923 19:33:29.025713 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:29.025720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:29.025745 (+ 25us) server_negotiation.cc:407] Connection header received
0923 19:33:29.025884 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:29.025890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:29.026046 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:29.026221 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:33:29.026774 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:29.027803 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:29.029424 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:29.029841 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:29.029949 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:29.030534 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:33:29.031192 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:33:29.031348 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:29.031408 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 19:33:29.031502 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260923 19:33:29.041335 16601 raft_consensus.cc:1261] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [term 1 FOLLOWER]: Refusing update from remote peer 4799dd750b3e46d2b46df4ab8e655115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:33:29.041478 16733 raft_consensus.cc:1261] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 1 FOLLOWER]: Refusing update from remote peer 4799dd750b3e46d2b46df4ab8e655115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:33:29.042484 16936 consensus_queue.cc:1035] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761512a07d6349c286a19261d3e33f56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:33:29.042946 16934 consensus_queue.cc:1035] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:33:29.055233 16939 mvcc.cc:205] Tried to move back new op lower bound from 7332626469021974528 to 7332626468800966656. Current Snapshot: MvccSnapshot[applied={T|T < 7332626469021974528}]
I20260923 19:33:29.060374 16941 mvcc.cc:205] Tried to move back new op lower bound from 7332626469021974528 to 7332626468800966656. Current Snapshot: MvccSnapshot[applied={T|T < 7332626469021974528}]
I20260923 19:33:29.070681 16460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:33:29.071100 16460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:33:29.075486 16460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Sending DeleteTablet for 3 replicas of tablet 144ae7878bf54028b87057067c251d7f
I20260923 19:33:29.077104 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 16498
I20260923 19:33:29.077010 16581 tablet_service.cc:1514] Processing DeleteTablet for tablet 144ae7878bf54028b87057067c251d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:33:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55730
I20260923 19:33:29.077293 16713 tablet_service.cc:1514] Processing DeleteTablet for tablet 144ae7878bf54028b87057067c251d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:33:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35446
I20260923 19:33:29.077912 16844 tablet_service.cc:1514] Processing DeleteTablet for tablet 144ae7878bf54028b87057067c251d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:33:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36158
I20260923 19:33:29.079890 16948 tablet_replica.cc:330] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: stopping tablet replica
I20260923 19:33:29.080633 16948 raft_consensus.cc:2229] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:33:29.083185 16948 raft_consensus.cc:2258] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:33:29.087409 16948 ts_tablet_manager.cc:1905] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:33:29.092700 16949 tablet_replica.cc:330] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: stopping tablet replica
I20260923 19:33:29.093345 16949 raft_consensus.cc:2229] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:33:29.093923 16949 raft_consensus.cc:2258] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:33:29.094851 16950 tablet_replica.cc:330] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: stopping tablet replica
I20260923 19:33:29.095830 16950 raft_consensus.cc:2229] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:33:29.097771 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.078319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.0.97.1:41667)
0923 19:33:29.078529 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:29.078535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:29.078560 (+ 25us) server_negotiation.cc:407] Connection header received
0923 19:33:29.078639 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:29.078645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:29.078771 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:29.078899 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:29.080884 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:29.082219 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:29.084473 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:29.085203 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:29.085347 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:29.088933 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:29.088949 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:29.088988 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:29.089041 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:29.093275 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:29.094225 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:29.094233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:29.094237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:29.094894 (+ 657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:29.095641 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:29.095647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:29.095650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:29.096250 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:29.096719 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:29.096943 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 19:33:29.097256 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 19:33:29.097851 16950 raft_consensus.cc:2258] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:33:29.098901 16641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:29.100032 16949 ts_tablet_manager.cc:1905] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:33:29.101105 16950 ts_tablet_manager.cc:1905] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:33:29.107092 16949 ts_tablet_manager.cc:1918] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:33:29.107484 16949 log.cc:1196] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/144ae7878bf54028b87057067c251d7f
I20260923 19:33:29.108070 16949 ts_tablet_manager.cc:1939] T 144ae7878bf54028b87057067c251d7f P 03d1f4dcbc1e4244baf8f1e4d98bb3c8: Deleting consensus metadata
I20260923 19:33:29.108611 16950 ts_tablet_manager.cc:1918] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:33:29.108842 16950 log.cc:1196] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/144ae7878bf54028b87057067c251d7f
I20260923 19:33:29.109376 16950 ts_tablet_manager.cc:1939] T 144ae7878bf54028b87057067c251d7f P 4799dd750b3e46d2b46df4ab8e655115: Deleting consensus metadata
I20260923 19:33:29.110750 16446 catalog_manager.cc:4841] TS 03d1f4dcbc1e4244baf8f1e4d98bb3c8 (127.0.97.2:39801): tablet 144ae7878bf54028b87057067c251d7f (table test-table [id=2b7a26c8014a4ecf815013033f6768cb]) successfully deleted
I20260923 19:33:29.111135 16446 catalog_manager.cc:4841] TS 4799dd750b3e46d2b46df4ab8e655115 (127.0.97.3:39237): tablet 144ae7878bf54028b87057067c251d7f (table test-table [id=2b7a26c8014a4ecf815013033f6768cb]) successfully deleted
I20260923 19:33:29.209843 16948 ts_tablet_manager.cc:1918] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:33:29.210101 16948 log.cc:1196] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/144ae7878bf54028b87057067c251d7f
I20260923 19:33:29.210701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 16498
I20260923 19:33:29.210753 16948 ts_tablet_manager.cc:1939] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Deleting consensus metadata
W20260923 19:33:29.227252 16445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41667 (error 108)
I20260923 19:33:29.228801 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 16650
I20260923 19:33:29.229102 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.228640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:55754)
0923 19:33:29.228840 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:29.228940 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 19:33:29.229377 16445 catalog_manager.cc:4584] TS 761512a07d6349c286a19261d3e33f56 (127.0.97.1:41667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 144ae7878bf54028b87057067c251d7f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41667: connect: Connection refused (error 111)
I20260923 19:33:29.246309 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.229813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.0.97.2:39801)
0923 19:33:29.230060 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:29.230067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:29.230088 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:33:29.230205 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:29.230211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:29.230339 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:29.230430 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:29.231666 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:29.232432 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:29.233678 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:29.234062 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:29.234165 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:29.240468 (+ 6303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:29.240484 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:29.240500 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:29.240537 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:29.243623 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:29.244301 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:29.244310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:29.244313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:29.244425 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:29.244767 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:29.244771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:29.244773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:29.245153 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:29.245419 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:29.245741 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 19:33:29.245959 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 19:33:29.246943 16773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:29.283561 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.282807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:55758)
0923 19:33:29.283113 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:29.283285 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 19:33:29.285110 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.284535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:55766)
0923 19:33:29.284732 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:29.284916 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 19:33:29.339211 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 16650
I20260923 19:33:29.346020 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.345372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:55772)
0923 19:33:29.345704 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:29.345833 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 19:33:29.347417 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.346928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:55780)
0923 19:33:29.347115 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:29.347240 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 19:33:29.354956 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 16781
I20260923 19:33:29.372663 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.355762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.0.97.3:39237)
0923 19:33:29.355941 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 19:33:29.355946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:33:29.355972 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:33:29.356247 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:33:29.356253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:33:29.356331 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:33:29.356430 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:33:29.358424 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:29.359422 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:29.361481 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:33:29.361995 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:33:29.362108 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:33:29.365880 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:33:29.365891 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:33:29.365916 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:33:29.365957 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:33:29.369433 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:29.370411 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:29.370419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:29.370421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:29.370592 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:33:29.371026 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:33:29.371029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:33:29.371031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:33:29.371391 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:33:29.371623 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 19:33:29.372094 (+ 471us) server_negotiation.cc:299] Negotiation successful
0923 19:33:29.372304 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 19:33:29.373667 16904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:29.428304 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.427705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:55786)
0923 19:33:29.427941 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:29.428040 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 19:33:29.429713 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.429279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:55802)
0923 19:33:29.429505 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:29.429603 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260923 19:33:29.498030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 16781
I20260923 19:33:29.517338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 16428
I20260923 19:33:29.518729 16491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:33:29.648088 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:29.647574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41667 (local address 127.0.0.1:55818)
0923 19:33:29.647803 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:29.647907 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 19:33:29.648075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 16428
I20260923 19:33:49.671162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38589
--webserver_interface=127.0.97.62
--webserver_port=35389
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:33:50.053743 16956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:50.054174 16956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:50.054368 16956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:50.062601 16956 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 19:33:50.062755 16956 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 19:33:50.062964 16956 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 19:33:50.063061 16956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:33:50.063138 16956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:50.063246 16956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:50.063304 16956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:33:50.063382 16956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:33:50.074836 16956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:50.076529 16956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:50.078262 16956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:50.092231 16962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:50.092160 16961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:50.092562 16956 server_base.cc:1034] running on GCE node
W20260923 19:33:50.092196 16964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:50.093668 16956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:50.095458 16956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:50.096701 16956 hybrid_clock.cc:648] HybridClock initialized: now 1790192030096629 us; error 44 us; skew 500 ppm
I20260923 19:33:50.097117 16956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:50 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:50.104488 16956 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:33:50.106331 16956 webserver.cc:460] Webserver started at http://127.0.97.62:35389/ using document root <none> and password file <none>
I20260923 19:33:50.107211 16956 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:50.107350 16956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:50.114568 16956 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.006s
I20260923 19:33:50.117929 16971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:50.119529 16956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 19:33:50.119709 16956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "e8d0f9b54bb54487991a3a8a6ce86e92"
format_stamp: "Formatted at 2026-09-23 19:33:03 on dist-test-slave-kxfb"
server_key: "0be2b5855f9094ad11e79f56fc254c31"
server_key_iv: "54ba1b0ab20ccced0fd86f06b55437f4"
server_key_version: "encryptionkey@0"
I20260923 19:33:50.120838 16956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:50.142454 16956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:50.147348 16956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:50.147692 16956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:50.166378 16956 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38589
I20260923 19:33:50.166400 17022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38589 every 8 connection(s)
I20260923 19:33:50.168126 16956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:33:50.173004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 16956
I20260923 19:33:50.174048 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41667
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38589
--webserver_port=38513
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:33:50.177174 17023 sys_catalog.cc:263] Verifying existing consensus state
I20260923 19:33:50.182188 17023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Bootstrap starting.
I20260923 19:33:50.206583 17023 log.cc:824] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Log is configured to *not* fsync() on all Append() calls
I20260923 19:33:50.225731 17023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 19:33:50.226392 17023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Bootstrap complete.
I20260923 19:33:50.235761 17023 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:50.236706 17023 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d0f9b54bb54487991a3a8a6ce86e92, State: Initialized, Role: FOLLOWER
I20260923 19:33:50.237373 17023 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:50.237632 17023 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:33:50.237769 17023 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:33:50.237943 17023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 2 FOLLOWER]: Advancing to term 3
I20260923 19:33:50.242013 17023 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:50.242601 17023 leader_election.cc:302] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8d0f9b54bb54487991a3a8a6ce86e92; no voters:
I20260923 19:33:50.243264 17023 leader_election.cc:288] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 19:33:50.243409 17027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 19:33:50.244781 17027 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [term 3 LEADER]: Becoming Leader. State: Replica: e8d0f9b54bb54487991a3a8a6ce86e92, State: Running, Role: LEADER
I20260923 19:33:50.245512 17027 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } }
I20260923 19:33:50.245756 17023 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:33:50.248301 17029 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8d0f9b54bb54487991a3a8a6ce86e92. Latest consensus state: current_term: 3 leader_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } } }
I20260923 19:33:50.248848 17029 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: This master's current role is: LEADER
I20260923 19:33:50.249722 17028 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d0f9b54bb54487991a3a8a6ce86e92" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38589 } } }
I20260923 19:33:50.253340 17028 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92 [sys.catalog]: This master's current role is: LEADER
I20260923 19:33:50.260975 17033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:33:50.270514 17033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:33:50.272285 17033 catalog_manager.cc:1259] Loaded cluster ID: 94dffc4be6f943229659164657bea261
I20260923 19:33:50.272382 17033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:33:50.276511 17033 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:33:50.278218 17033 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Loaded TSK: 0
I20260923 19:33:50.278610 17033 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e8d0f9b54bb54487991a3a8a6ce86e92: Loaded TSK: 1
F20260923 19:33:50.279881 17033 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe5de1b29ad google::LogMessage::Fail() at ??:0
@ 0x7fe5de1b588a google::LogMessage::SendToLog() at ??:0
@ 0x7fe5de1b24de google::LogMessage::Flush() at ??:0
@ 0x7fe5de1b3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe5e1f167d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe5e1f0573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe5f3be11c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe5f3be0783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe5f3c6c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe5f3c6c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe5f3cb65e9 std::function<>::operator()() at ??:0
@ 0x7fe5f3be4cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe5f3be2c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe5f3c6a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe5f3c6a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe5e0dfe345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe5e0e0191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe5e0e01799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe5e0ddceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe5e40b66db start_thread at ??:0
@ 0x7fe5dd04c61f clone at ??:0
W20260923 19:33:50.596128 17025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:50.596490 17025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:50.596630 17025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:50.604507 17025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:50.604674 17025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:50.604846 17025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:33:50.615880 17025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38513
--tserver_master_addrs=127.0.97.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:50.617591 17025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:50.619249 17025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:50.634354 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:50.634436 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:50.634366 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:50.635411 17025 server_base.cc:1034] running on GCE node
I20260923 19:33:50.636152 17025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:50.637257 17025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:50.638465 17025 hybrid_clock.cc:648] HybridClock initialized: now 1790192030638397 us; error 75 us; skew 500 ppm
I20260923 19:33:50.638816 17025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:50 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:50.642089 17025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:33:50.643787 17025 webserver.cc:460] Webserver started at http://127.0.97.1:38513/ using document root <none> and password file <none>
I20260923 19:33:50.644558 17025 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:50.644668 17025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:50.650774 17025 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 19:33:50.654036 17080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:50.655830 17025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:33:50.656023 17025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "761512a07d6349c286a19261d3e33f56"
format_stamp: "Formatted at 2026-09-23 19:33:04 on dist-test-slave-kxfb"
server_key: "ded2b107b70fe8a0767c7571644f0392"
server_key_iv: "604c8114e81e9a5c1cd4e6c24c761bdb"
server_key_version: "encryptionkey@0"
I20260923 19:33:50.657099 17025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:50.679164 17025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:50.683463 17025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:50.683696 17025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:50.684751 17025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:33:50.686744 17087 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260923 19:33:50.693065 17087 ts_tablet_manager.cc:1852] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 19:33:50.694185 17087 ts_tablet_manager.cc:1905] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:33:50.702585 17087 ts_tablet_manager.cc:1918] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:33:50.702875 17087 ts_tablet_manager.cc:1939] T 144ae7878bf54028b87057067c251d7f P 761512a07d6349c286a19261d3e33f56: Deleting consensus metadata
I20260923 19:33:50.703385 17025 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 19:33:50.703562 17025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260923 19:33:50.703686 17025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:33:50.703764 17025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:50.739619 17025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41667
I20260923 19:33:50.739718 17193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41667 every 8 connection(s)
I20260923 19:33:50.740986 17025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:33:50.746634 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:50.744416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:53277)
0923 19:33:50.745309 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:50.745537 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":324,"threads_started":1}
W20260923 19:33:50.747530 17194 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38589 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38589: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
I20260923 19:33:50.748047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 17025
I20260923 19:33:50.748733 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:50.748255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:38803)
0923 19:33:50.748464 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:50.748568 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 19:33:50.748924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39801
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38589
--webserver_port=44165
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:33:50.750000 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:50.749485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:51453)
0923 19:33:50.749652 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:50.749799 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 19:33:50.750346 17194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 19:33:51.091997 17197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:51.092283 17197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:51.092394 17197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:51.099586 17197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:51.099694 17197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:51.099836 17197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:33:51.109472 17197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44165
--tserver_master_addrs=127.0.97.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:51.110858 17197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:51.112140 17197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:51.126381 17202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:51.126417 17203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:51.126400 17205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:51.127683 17197 server_base.cc:1034] running on GCE node
I20260923 19:33:51.128312 17197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:51.129261 17197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:51.130457 17197 hybrid_clock.cc:648] HybridClock initialized: now 1790192031130405 us; error 43 us; skew 500 ppm
I20260923 19:33:51.130780 17197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:51 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:51.133888 17197 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:33:51.135423 17197 webserver.cc:460] Webserver started at http://127.0.97.2:44165/ using document root <none> and password file <none>
I20260923 19:33:51.136119 17197 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:51.136238 17197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:51.141750 17197 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260923 19:33:51.144886 17212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:51.146515 17197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:33:51.146697 17197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "03d1f4dcbc1e4244baf8f1e4d98bb3c8"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "7934e8b7fdeaab0822df49125ed9866e"
server_key_iv: "a64699d5d1b568481f550c4da131a679"
server_key_version: "encryptionkey@0"
I20260923 19:33:51.147750 17197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:51.175597 17197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:51.179812 17197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:51.180071 17197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:51.181149 17197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:33:51.182564 17197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:33:51.182667 17197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:51.182773 17197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:33:51.182844 17197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:51.217270 17197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39801
I20260923 19:33:51.217378 17324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39801 every 8 connection(s)
I20260923 19:33:51.218776 17197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:33:51.224434 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:51.222185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:41503)
0923 19:33:51.223059 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:51.223319 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":301,"threads_started":1}
W20260923 19:33:51.225304 17325 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38589 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38589: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
I20260923 19:33:51.226379 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:51.225976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:48805)
0923 19:33:51.226145 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:51.226224 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 19:33:51.226429 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 17197
I20260923 19:33:51.227176 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39237
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38589
--webserver_port=34577
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:33:51.227687 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:51.227156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:43235)
0923 19:33:51.227403 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:51.227534 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260923 19:33:51.228061 17325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 19:33:51.545404 17328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:33:51.545693 17328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:33:51.545786 17328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:33:51.553164 17328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:33:51.553282 17328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:33:51.553395 17328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:33:51.563230 17328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34577
--tserver_master_addrs=127.0.97.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:33:51.564461 17328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:33:51.565819 17328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:33:51.579439 17334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:51.579537 17336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:33:51.579473 17333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:33:51.581053 17328 server_base.cc:1034] running on GCE node
I20260923 19:33:51.581692 17328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:33:51.582736 17328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:33:51.583973 17328 hybrid_clock.cc:648] HybridClock initialized: now 1790192031583911 us; error 54 us; skew 500 ppm
I20260923 19:33:51.584301 17328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:33:51 dist-test-slave-kxfb krb5kdc[15876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:33:51.587714 17328 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:33:51.589373 17328 webserver.cc:460] Webserver started at http://127.0.97.3:34577/ using document root <none> and password file <none>
I20260923 19:33:51.590071 17328 fs_manager.cc:359] Metadata directory not provided
I20260923 19:33:51.590188 17328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:33:51.596304 17328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:33:51.599896 17343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:51.601881 17328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 19:33:51.602075 17328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "4799dd750b3e46d2b46df4ab8e655115"
format_stamp: "Formatted at 2026-09-23 19:33:05 on dist-test-slave-kxfb"
server_key: "bca995437577b8777882bc8bcd645115"
server_key_iv: "b5a2a74b6ed5f61580ef50cf1c158b16"
server_key_version: "encryptionkey@0"
I20260923 19:33:51.603410 17328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:33:51.633622 17328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:33:51.637609 17328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:33:51.637846 17328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:33:51.638963 17328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:33:51.640357 17328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:33:51.640496 17328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:51.640689 17328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:33:51.640800 17328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:33:51.674957 17328 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39237
I20260923 19:33:51.675096 17455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39237 every 8 connection(s)
I20260923 19:33:51.676373 17328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:33:51.681169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 17328
I20260923 19:33:51.682130 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:51.679963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:47313)
0923 19:33:51.680825 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:51.681091 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":323,"threads_started":1}
W20260923 19:33:51.682911 17456 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38589 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38589: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
I20260923 19:33:51.683804 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:51.683467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:34193)
0923 19:33:51.683620 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:51.683695 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 19:33:51.684855 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:51.684535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:58121)
0923 19:33:51.684678 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:51.684752 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 19:33:51.685081 17456 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 19:33:51.752235 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:51.751166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:60309)
0923 19:33:51.751860 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:51.751982 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":348,"threads_started":1}
I20260923 19:33:52.229687 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:52.228723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:53391)
0923 19:33:52.229311 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:52.229429 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":275,"threads_started":1}
I20260923 19:33:52.686762 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:52.685790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:59275)
0923 19:33:52.686368 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:52.686491 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":260,"threads_started":1}
I20260923 19:33:52.754210 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:52.753375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:47631)
0923 19:33:52.753907 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:52.754004 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":292,"threads_started":1}
I20260923 19:33:53.231858 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:53.230916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:59221)
0923 19:33:53.231493 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:53.231591 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":282,"threads_started":1}
I20260923 19:33:53.688619 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:53.687777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:36701)
0923 19:33:53.688320 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:53.688411 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":277,"threads_started":1}
I20260923 19:33:53.756172 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:53.755302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:33009)
0923 19:33:53.755876 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:53.755973 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":279,"threads_started":1}
I20260923 19:33:54.233666 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:54.232784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:44141)
0923 19:33:54.233360 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:54.233454 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":312,"threads_started":1}
I20260923 19:33:54.690583 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:54.689693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:33845)
0923 19:33:54.690220 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:54.690365 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":280,"threads_started":1}
I20260923 19:33:54.758172 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:54.757286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:48347)
0923 19:33:54.757857 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:54.757951 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":314,"threads_started":1}
I20260923 19:33:55.235561 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:55.234725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:45395)
0923 19:33:55.235253 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:55.235350 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":261,"threads_started":1}
I20260923 19:33:55.692298 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:55.691507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:44643)
0923 19:33:55.692006 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:55.692088 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
I20260923 19:33:55.760159 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:55.759322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:39061)
0923 19:33:55.759863 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:55.759954 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":290,"threads_started":1}
I20260923 19:33:56.237622 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:56.236734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:44785)
0923 19:33:56.237297 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:56.237405 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":293,"threads_started":1}
I20260923 19:33:56.694155 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:56.693304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:57035)
0923 19:33:56.693849 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:56.693942 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":306,"threads_started":1}
I20260923 19:33:56.762249 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:56.761221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:41139)
0923 19:33:56.761887 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:56.761999 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":403,"threads_started":1}
I20260923 19:33:57.239432 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:57.238578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:60901)
0923 19:33:57.239109 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:57.239235 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":282,"threads_started":1}
I20260923 19:33:57.696092 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:57.695207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:51255)
0923 19:33:57.695790 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:57.695891 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":331,"threads_started":1}
I20260923 19:33:57.764192 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:57.763258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:40421)
0923 19:33:57.763859 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:57.763951 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":282,"threads_started":1}
I20260923 19:33:58.241598 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:58.240632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:55875)
0923 19:33:58.241250 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:58.241341 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":271,"threads_started":1}
I20260923 19:33:58.698203 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:58.697221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:38881)
0923 19:33:58.697842 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:58.697943 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":308,"threads_started":1}
I20260923 19:33:58.766040 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:58.765210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:39969)
0923 19:33:58.765723 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:58.765812 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":247,"threads_started":1}
I20260923 19:33:59.243731 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:59.242857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:36853)
0923 19:33:59.243433 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:59.243529 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":316,"threads_started":1}
I20260923 19:33:59.700096 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:59.699287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:44045)
0923 19:33:59.699800 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:59.699884 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":271,"threads_started":1}
I20260923 19:33:59.767829 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:33:59.766978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:36541)
0923 19:33:59.767538 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0923 19:33:59.767641 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":258,"threads_started":1}
I20260923 19:34:00.245579 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:00.244745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:48805)
0923 19:34:00.245275 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:00.245362 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":281,"threads_started":1}
I20260923 19:34:00.702232 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:00.701210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:44815)
0923 19:34:00.701867 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:00.701992 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":336,"threads_started":1}
I20260923 19:34:00.769675 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:00.768849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:40323)
0923 19:34:00.769352 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:00.769453 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":262,"threads_started":1}
I20260923 19:34:01.247618 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:01.246773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:46921)
0923 19:34:01.247290 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:01.247388 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
I20260923 19:34:01.704165 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:01.703288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:55123)
0923 19:34:01.703853 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:01.703938 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":330,"threads_started":1}
I20260923 19:34:01.771759 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:01.770836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:45273)
0923 19:34:01.771423 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:01.771518 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":326,"threads_started":1}
I20260923 19:34:02.249603 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:02.248713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:53135)
0923 19:34:02.249279 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:02.249376 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":310,"threads_started":1}
I20260923 19:34:02.706001 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:02.705179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:57409)
0923 19:34:02.705685 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:02.705784 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":273,"threads_started":1}
I20260923 19:34:02.773645 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:02.772709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:34159)
0923 19:34:02.773281 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:02.773413 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":312,"threads_started":1}
I20260923 19:34:03.251479 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:03.250643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:41671)
0923 19:34:03.251167 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:03.251266 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":268,"threads_started":1}
I20260923 19:34:03.708050 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:03.707060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:38549)
0923 19:34:03.707611 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:03.707721 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":317,"threads_started":1}
I20260923 19:34:03.775625 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:03.774737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:56323)
0923 19:34:03.775272 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:03.775385 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":278,"threads_started":1}
I20260923 19:34:04.253479 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:04.252594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:42367)
0923 19:34:04.253151 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:04.253250 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":283,"threads_started":1}
I20260923 19:34:04.710088 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:04.709083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:49401)
0923 19:34:04.709751 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:04.709852 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":399,"threads_started":1}
I20260923 19:34:04.777606 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:04.776740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:49727)
0923 19:34:04.777288 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:04.777375 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":301,"threads_started":1}
I20260923 19:34:05.255308 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:05.254465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:40213)
0923 19:34:05.254978 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:05.255071 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":244,"threads_started":1}
I20260923 19:34:05.712028 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:05.711201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:42727)
0923 19:34:05.711712 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:05.711800 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":276,"threads_started":1}
I20260923 19:34:05.779891 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:05.779025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.1:53957)
0923 19:34:05.779580 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:05.779670 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
I20260923 19:34:06.257405 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:06.256541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.2:37755)
0923 19:34:06.257054 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:06.257151 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":259,"threads_started":1}
I20260923 19:34:06.684671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 17025
I20260923 19:34:06.708393 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:06.686012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.0.97.1:41667)
0923 19:34:06.686710 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:06.686732 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:06.686781 (+ 49us) server_negotiation.cc:407] Connection header received
0923 19:34:06.688698 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:06.688711 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:06.689268 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:06.689654 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:06.691338 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:06.692704 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:06.694021 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:06.694860 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:06.695041 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:06.701535 (+ 6494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:06.701545 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:06.701568 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:06.701618 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:06.704714 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:06.705430 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:06.705438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:06.705446 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:06.705628 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:06.705999 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:06.706002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:06.706004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:06.706700 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:06.707028 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:06.707519 (+ 491us) server_negotiation.cc:299] Negotiation successful
0923 19:34:06.708057 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":348,"threads_started":1}
I20260923 19:34:06.710171 17189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:06.715250 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:06.713363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38589 (local address 127.0.97.3:35119)
0923 19:34:06.714856 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:06.714978 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":1195,"threads_started":1}
I20260923 19:34:06.806813 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 17025
I20260923 19:34:06.823856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 17197
I20260923 19:34:06.844179 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:06.824957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.97.2:39801)
0923 19:34:06.825591 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:06.825609 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:06.825653 (+ 44us) server_negotiation.cc:407] Connection header received
0923 19:34:06.827574 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:06.827588 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:06.828146 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:06.828549 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:06.830246 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:06.831811 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:06.833222 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:06.833996 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:06.834203 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:06.837336 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:06.837347 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:06.837366 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:06.837420 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:06.840613 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:06.841335 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:06.841343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:06.841350 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:06.841528 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:06.841901 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:06.841905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:06.841907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:06.842559 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:06.842875 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:06.843372 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 19:34:06.843832 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":315,"threads_started":1}
I20260923 19:34:06.845834 17320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:06.935272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 17197
I20260923 19:34:06.954533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 17328
I20260923 19:34:06.973088 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:06.955605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.0.97.3:39237)
0923 19:34:06.956272 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:06.956292 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:06.956342 (+ 50us) server_negotiation.cc:407] Connection header received
0923 19:34:06.958192 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:06.958202 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:06.958916 (+ 714us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:06.959251 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:06.960629 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:06.961955 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:06.963296 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:06.964024 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:06.964169 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:06.966940 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:06.966949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:06.966968 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:06.967010 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:06.969811 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:06.970551 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:06.970558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:06.970565 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:06.970735 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:06.971056 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:06.971059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:06.971061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:06.971676 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:06.971938 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:06.972282 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 19:34:06.972732 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":306,"threads_started":1}
I20260923 19:34:06.974783 17451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:07.064374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 17328
2026-09-23T19:34:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64966 ms)
[----------] 30 tests from SecurityITest (302745 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:07 dist-test-slave-kxfb krb5kdc[17514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:07 dist-test-slave-kxfb krb5kdc[17514](info): set up 2 sockets
Sep 23 19:34:07 dist-test-slave-kxfb krb5kdc[17514](info): commencing operation
krb5kdc: starting...
W20260923 19:34:08.102679 388 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:08 dist-test-slave-kxfb krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:34:08.374331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33455
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:08.740828 17527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:08.741189 17527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:08.741282 17527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:08.748595 17527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:08.748710 17527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:08.748757 17527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:08.748822 17527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:08.748876 17527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:08.758661 17527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:08.759958 17527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:08.761235 17527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:08.773751 17532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:08.773854 17533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:08.773923 17535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:08.774740 17527 server_base.cc:1034] running on GCE node
I20260923 19:34:08.775583 17527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:08.777037 17527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:08.778319 17527 hybrid_clock.cc:648] HybridClock initialized: now 1790192048778223 us; error 58 us; skew 500 ppm
I20260923 19:34:08.778708 17527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:08 dist-test-slave-kxfb krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:08.782071 17527 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:34:08.783699 17527 webserver.cc:460] Webserver started at http://127.0.97.62:34951/ using document root <none> and password file <none>
I20260923 19:34:08.784395 17527 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:08.784498 17527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:08.784792 17527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:08.787833 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "1c7fa66b211649a28ae1214808f746fc"
format_stamp: "Formatted at 2026-09-23 19:34:08 on dist-test-slave-kxfb"
server_key: "f76ad6c98ac1e193d6bd49697de98e1b"
server_key_iv: "b6c3fc1236a0ecf5a5cd5ee894a070b4"
server_key_version: "encryptionkey@0"
I20260923 19:34:08.788568 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c7fa66b211649a28ae1214808f746fc"
format_stamp: "Formatted at 2026-09-23 19:34:08 on dist-test-slave-kxfb"
server_key: "f76ad6c98ac1e193d6bd49697de98e1b"
server_key_iv: "b6c3fc1236a0ecf5a5cd5ee894a070b4"
server_key_version: "encryptionkey@0"
I20260923 19:34:08.794147 17527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 19:34:08.798117 17542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:08.799827 17527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:34:08.800058 17527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "1c7fa66b211649a28ae1214808f746fc"
format_stamp: "Formatted at 2026-09-23 19:34:08 on dist-test-slave-kxfb"
server_key: "f76ad6c98ac1e193d6bd49697de98e1b"
server_key_iv: "b6c3fc1236a0ecf5a5cd5ee894a070b4"
server_key_version: "encryptionkey@0"
I20260923 19:34:08.800282 17527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:08.823535 17527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:08.827937 17527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:08.828217 17527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:08.847492 17527 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33455
I20260923 19:34:08.847568 17593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33455 every 8 connection(s)
I20260923 19:34:08.848870 17527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:08.851125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 17527
I20260923 19:34:08.851320 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:08.851948 388 external_mini_cluster.cc:1428] Setting key dd40fce3a0ebcbb9fc97634357c3a431
I20260923 19:34:08.853906 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 19:34:08 dist-test-slave-kxfb krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:08.869199 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc: Bootstrap starting.
I20260923 19:34:08.873857 17594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:08.875360 17594 log.cc:824] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:08.878252 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:08.854099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.0.97.62:33455)
0923 19:34:08.855054 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:08.855072 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:08.855132 (+ 60us) server_negotiation.cc:407] Connection header received
0923 19:34:08.857476 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:08.857508 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:08.858163 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:08.858782 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:08.860091 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:08.862243 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:08.863811 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:08.864953 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:08.865173 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:08.869217 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:08.869238 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:08.869268 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:08.869331 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:08.873216 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:08.873944 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:08.873954 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:08.873962 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:08.874191 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:08.874696 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:08.874700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:08.874703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:08.875523 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:08.875829 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:08.876322 (+ 493us) server_negotiation.cc:299] Negotiation successful
0923 19:34:08.876817 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":405,"threads_started":1}
I20260923 19:34:08.879487 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc: No bootstrap required, opened a new log
I20260923 19:34:08.888254 17594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7fa66b211649a28ae1214808f746fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33455 } }
I20260923 19:34:08.888760 17594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:08.888892 17594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7fa66b211649a28ae1214808f746fc, State: Initialized, Role: FOLLOWER
I20260923 19:34:08.889586 17594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7fa66b211649a28ae1214808f746fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33455 } }
I20260923 19:34:08.889839 17594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:08.889959 17594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:08.890101 17594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:08.892584 17594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7fa66b211649a28ae1214808f746fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33455 } }
I20260923 19:34:08.893158 17594 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c7fa66b211649a28ae1214808f746fc; no voters:
I20260923 19:34:08.893827 17594 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:08.894083 17599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:08.895097 17599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [term 1 LEADER]: Becoming Leader. State: Replica: 1c7fa66b211649a28ae1214808f746fc, State: Running, Role: LEADER
I20260923 19:34:08.895793 17599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7fa66b211649a28ae1214808f746fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33455 } }
I20260923 19:34:08.896453 17594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:08.899461 17600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c7fa66b211649a28ae1214808f746fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7fa66b211649a28ae1214808f746fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33455 } } }
I20260923 19:34:08.899590 17601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7fa66b211649a28ae1214808f746fc. Latest consensus state: current_term: 1 leader_uuid: "1c7fa66b211649a28ae1214808f746fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7fa66b211649a28ae1214808f746fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33455 } } }
I20260923 19:34:08.899942 17600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:08.900032 17601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:08.902328 17605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:08.908838 17605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:08.919358 17605 catalog_manager.cc:1347] Generated new cluster ID: 6210447e76644083bbf8ee270b8ab181
I20260923 19:34:08.919487 17605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:08.934046 17605 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:08.935680 17605 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:08.944370 17605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc: Generated new TSK 0
I20260923 19:34:08.945541 17605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:34:09.016121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33455
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:09.391600 17620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:09.391976 17620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:09.392118 17620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:09.400512 17620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:09.400676 17620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:09.400887 17620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:34:09.412258 17620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:09.413946 17620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:09.415566 17620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:09.430411 17626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:09.430495 17625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:09.430491 17628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:09.431838 17620 server_base.cc:1034] running on GCE node
I20260923 19:34:09.432626 17620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:09.433794 17620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:09.435026 17620 hybrid_clock.cc:648] HybridClock initialized: now 1790192049434950 us; error 66 us; skew 500 ppm
I20260923 19:34:09.435408 17620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:09 dist-test-slave-kxfb krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:09.439004 17620 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:34:09.440922 17620 webserver.cc:460] Webserver started at http://127.0.97.1:34215/ using document root <none> and password file <none>
I20260923 19:34:09.441818 17620 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:09.441994 17620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:09.442458 17620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:09.446102 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "dd33d746d89643d79cd78692a9dd6847"
format_stamp: "Formatted at 2026-09-23 19:34:09 on dist-test-slave-kxfb"
server_key: "6634962820de6bffd6fca9606f85d77e"
server_key_iv: "9d414ee67b910cf72c2b3de7b6500012"
server_key_version: "encryptionkey@0"
I20260923 19:34:09.447026 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dd33d746d89643d79cd78692a9dd6847"
format_stamp: "Formatted at 2026-09-23 19:34:09 on dist-test-slave-kxfb"
server_key: "6634962820de6bffd6fca9606f85d77e"
server_key_iv: "9d414ee67b910cf72c2b3de7b6500012"
server_key_version: "encryptionkey@0"
I20260923 19:34:09.452973 17620 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260923 19:34:09.456991 17635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:09.458640 17620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:34:09.458837 17620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "dd33d746d89643d79cd78692a9dd6847"
format_stamp: "Formatted at 2026-09-23 19:34:09 on dist-test-slave-kxfb"
server_key: "6634962820de6bffd6fca9606f85d77e"
server_key_iv: "9d414ee67b910cf72c2b3de7b6500012"
server_key_version: "encryptionkey@0"
I20260923 19:34:09.459056 17620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:09.480656 17620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:09.485425 17620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:09.485771 17620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:09.486951 17620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:09.488476 17620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:09.488557 17620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:09.488694 17620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:09.488749 17620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:09.524782 17620 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42227
I20260923 19:34:09.524839 17747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42227 every 8 connection(s)
I20260923 19:34:09.526259 17620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:34:09.536015 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 17620
I20260923 19:34:09.536202 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:34:09.536815 388 external_mini_cluster.cc:1428] Setting key 4c1ebc020af441d5fcd6834a45affd54
Sep 23 19:34:09 dist-test-slave-kxfb krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:09.549358 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:09.529146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56143 (local address 127.0.97.62:33455)
0923 19:34:09.529676 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:09.529681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:09.530690 (+ 1009us) server_negotiation.cc:407] Connection header received
0923 19:34:09.532856 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:09.532862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:09.532958 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:09.533064 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:09.534867 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:09.535681 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:09.537462 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:09.538076 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:09.538214 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:09.542596 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:09.542603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:09.542608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:09.542659 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:09.545819 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:09.546754 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:09.546760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:09.546763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:09.546911 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:09.547407 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:09.547412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:09.547415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:09.547756 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:09.547901 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:09.548902 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0923 19:34:09.549092 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":253,"threads_started":1}
I20260923 19:34:09.551100 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:09.529642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33455 (local address 127.0.97.1:56143)
0923 19:34:09.530501 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:09.530554 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:09.532536 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:09.533367 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:09.533377 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:09.533973 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:09.534690 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:09.534709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:09.535835 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:09.535839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:09.537253 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:09.537276 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:09.538391 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:09.538400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:09.538739 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:09.539644 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:34:09.539675 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:09.542250 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:09.546004 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:09.546014 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:09.546031 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:09.546603 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:09.547032 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:09.547035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:09.547038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:09.547291 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:09.547904 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:09.547911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:09.548442 (+ 531us) client_negotiation.cc:769] Sending connection context
0923 19:34:09.548798 (+ 356us) client_negotiation.cc:240] Negotiation successful
0923 19:34:09.549489 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":297,"threads_started":1}
I20260923 19:34:09.552847 17748 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33455
I20260923 19:34:09.553185 17748 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:09.553992 17748 heartbeater.cc:511] Master 127.0.97.62:33455 requested a full tablet report, sending...
I20260923 19:34:09.556107 17559 ts_manager.cc:194] Registered new tserver with Master: dd33d746d89643d79cd78692a9dd6847 (127.0.97.1:42227)
I20260923 19:34:09.557585 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56143
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:34:09.598145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33455
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:09.960350 17754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:09.960657 17754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:09.960753 17754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:09.968117 17754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:09.968238 17754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:09.968353 17754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:34:09.978513 17754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:09.979871 17754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:09.981233 17754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:09.995683 17759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:09.995674 17760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:09.995639 17762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:09.996879 17754 server_base.cc:1034] running on GCE node
I20260923 19:34:09.997567 17754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:09.998761 17754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:09.999996 17754 hybrid_clock.cc:648] HybridClock initialized: now 1790192049999931 us; error 60 us; skew 500 ppm
I20260923 19:34:10.000351 17754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:09 dist-test-slave-kxfb krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:10.003646 17754 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:34:10.005414 17754 webserver.cc:460] Webserver started at http://127.0.97.2:42041/ using document root <none> and password file <none>
I20260923 19:34:10.006137 17754 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:10.006251 17754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:10.006577 17754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:10.009570 17754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "ade0569071d840a8be88370770a142ea"
format_stamp: "Formatted at 2026-09-23 19:34:10 on dist-test-slave-kxfb"
server_key: "27ae7480f7de09512089034624d81a07"
server_key_iv: "c392c892801e1ee4e3f16991a77d0e30"
server_key_version: "encryptionkey@0"
I20260923 19:34:10.010483 17754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ade0569071d840a8be88370770a142ea"
format_stamp: "Formatted at 2026-09-23 19:34:10 on dist-test-slave-kxfb"
server_key: "27ae7480f7de09512089034624d81a07"
server_key_iv: "c392c892801e1ee4e3f16991a77d0e30"
server_key_version: "encryptionkey@0"
I20260923 19:34:10.016106 17754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 19:34:10.020004 17769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:10.021559 17754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:34:10.021744 17754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "ade0569071d840a8be88370770a142ea"
format_stamp: "Formatted at 2026-09-23 19:34:10 on dist-test-slave-kxfb"
server_key: "27ae7480f7de09512089034624d81a07"
server_key_iv: "c392c892801e1ee4e3f16991a77d0e30"
server_key_version: "encryptionkey@0"
I20260923 19:34:10.021926 17754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:10.033761 17754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:10.037763 17754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:10.038026 17754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:10.039076 17754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:10.040472 17754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:10.040548 17754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:10.040678 17754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:10.040729 17754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:10.075301 17754 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42355
I20260923 19:34:10.075400 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42355 every 8 connection(s)
I20260923 19:34:10.076581 17754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:34:10.077208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 17754
I20260923 19:34:10.077401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:34:10.077958 388 external_mini_cluster.cc:1428] Setting key 0d845eaaddf4237b0aa3296c0ef2302d
Sep 23 19:34:10 dist-test-slave-kxfb krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:10.098479 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.079785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41765 (local address 127.0.97.62:33455)
0923 19:34:10.080319 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.080324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.081432 (+ 1108us) server_negotiation.cc:407] Connection header received
0923 19:34:10.083726 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.083732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.083818 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.083924 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:10.085353 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.086074 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.087571 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.087977 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.088080 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.091928 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:10.091935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:10.091939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:10.091990 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:10.095243 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:10.096020 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:10.096026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:10.096030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:10.096221 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:10.096603 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:10.096607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:10.096609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:10.096976 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:10.097120 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.097816 (+ 696us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.098062 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":245,"threads_started":1}
I20260923 19:34:10.099205 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.080397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33455 (local address 127.0.97.2:41765)
0923 19:34:10.081250 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:10.081304 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:10.083333 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:10.084165 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:10.084173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:10.084656 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:10.085191 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.085207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.086218 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.086222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.087437 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.087449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.088184 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.088191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.088399 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.089002 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:34:10.089024 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:10.091587 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:10.095427 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:10.095433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:10.095445 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:10.095875 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:10.096325 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:10.096328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:10.096330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:10.096501 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:10.097112 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:10.097119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:10.097449 (+ 330us) client_negotiation.cc:769] Sending connection context
0923 19:34:10.097671 (+ 222us) client_negotiation.cc:240] Negotiation successful
0923 19:34:10.098052 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":338,"threads_started":1}
I20260923 19:34:10.100674 17882 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33455
I20260923 19:34:10.100963 17882 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:10.101694 17882 heartbeater.cc:511] Master 127.0.97.62:33455 requested a full tablet report, sending...
I20260923 19:34:10.103358 17559 ts_manager.cc:194] Registered new tserver with Master: ade0569071d840a8be88370770a142ea (127.0.97.2:42355)
I20260923 19:34:10.104297 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41765
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:34:10.141603 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33455
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:10.461534 17888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:10.461815 17888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:10.461908 17888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:10.469084 17888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:10.469197 17888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:10.469305 17888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:34:10.478850 17888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:10.480031 17888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:10.481217 17888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:10.493943 17893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:10.494021 17896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:10.494071 17894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:10.495946 17888 server_base.cc:1034] running on GCE node
I20260923 19:34:10.496520 17888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:10.497512 17888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:10.498708 17888 hybrid_clock.cc:648] HybridClock initialized: now 1790192050498648 us; error 60 us; skew 500 ppm
I20260923 19:34:10.499006 17888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:10 dist-test-slave-kxfb krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:10.502112 17888 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:34:10.503913 17888 webserver.cc:460] Webserver started at http://127.0.97.3:33879/ using document root <none> and password file <none>
I20260923 19:34:10.504594 17888 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:10.504707 17888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:10.505002 17888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:10.507887 17888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "089749e01eb44708a994a5447523cbcd"
format_stamp: "Formatted at 2026-09-23 19:34:10 on dist-test-slave-kxfb"
server_key: "dff50cd4a296c20df4b6e6397e1bc904"
server_key_iv: "782103351e3d05f868adb59cc3660651"
server_key_version: "encryptionkey@0"
I20260923 19:34:10.508651 17888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "089749e01eb44708a994a5447523cbcd"
format_stamp: "Formatted at 2026-09-23 19:34:10 on dist-test-slave-kxfb"
server_key: "dff50cd4a296c20df4b6e6397e1bc904"
server_key_iv: "782103351e3d05f868adb59cc3660651"
server_key_version: "encryptionkey@0"
I20260923 19:34:10.513847 17888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 19:34:10.517531 17903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:10.519117 17888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:34:10.519279 17888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "089749e01eb44708a994a5447523cbcd"
format_stamp: "Formatted at 2026-09-23 19:34:10 on dist-test-slave-kxfb"
server_key: "dff50cd4a296c20df4b6e6397e1bc904"
server_key_iv: "782103351e3d05f868adb59cc3660651"
server_key_version: "encryptionkey@0"
I20260923 19:34:10.519467 17888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:10.550820 17888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:10.554952 17888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:10.555217 17888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:10.556253 17888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:10.557619 17888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:10.557698 17888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:10.557816 17888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:10.557876 17888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:10.560768 17748 heartbeater.cc:503] Master 127.0.97.62:33455 was elected leader, sending a full tablet report...
I20260923 19:34:10.593050 17888 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42303
I20260923 19:34:10.593124 18015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42303 every 8 connection(s)
I20260923 19:34:10.594337 17888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:34:10.599064 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 17888
I20260923 19:34:10.599280 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:34:10.599887 388 external_mini_cluster.cc:1428] Setting key f5df26fe88bce827de9ccc135431e32e
Sep 23 19:34:10 dist-test-slave-kxfb krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:10.618065 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.597599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45997 (local address 127.0.97.62:33455)
0923 19:34:10.597855 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.597859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.599264 (+ 1405us) server_negotiation.cc:407] Connection header received
0923 19:34:10.601632 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.601638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.601752 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.601872 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:10.603741 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.604785 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.606454 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.606910 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.607023 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.610996 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:10.611003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:10.611006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:10.611051 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:10.614099 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:10.615248 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:10.615254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:10.615257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:10.615503 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:10.616082 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:10.616087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:10.616090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:10.616507 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:10.616648 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.617566 (+ 918us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.617802 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260923 19:34:10.619306 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.598130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33455 (local address 127.0.97.3:45997)
0923 19:34:10.599068 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:10.599123 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:10.601340 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:10.602233 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:10.602242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:10.602847 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:10.603546 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.603568 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.604979 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.604987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.606241 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.606254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.607116 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.607121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.607321 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.608039 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:34:10.608063 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:10.610707 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:10.614317 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:10.614331 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:10.614354 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:10.615061 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:10.615641 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:10.615646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:10.615650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:10.615921 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:10.616653 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:10.616660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:10.617108 (+ 448us) client_negotiation.cc:769] Sending connection context
0923 19:34:10.617413 (+ 305us) client_negotiation.cc:240] Negotiation successful
0923 19:34:10.617930 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":401,"threads_started":1}
I20260923 19:34:10.620940 18016 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33455
I20260923 19:34:10.621265 18016 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:10.622118 18016 heartbeater.cc:511] Master 127.0.97.62:33455 requested a full tablet report, sending...
I20260923 19:34:10.623843 17559 ts_manager.cc:194] Registered new tserver with Master: 089749e01eb44708a994a5447523cbcd (127.0.97.3:42303)
I20260923 19:34:10.624694 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45997
I20260923 19:34:10.628863 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:34:10.648038 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.634068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.0.97.62:33455)
0923 19:34:10.634373 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.634380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.634759 (+ 379us) server_negotiation.cc:407] Connection header received
0923 19:34:10.634965 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.634971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.635048 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.635137 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:10.636783 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.637595 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.639024 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.639465 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.639591 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.642174 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:10.642180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:10.642183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:10.642223 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:10.645049 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:10.645793 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:10.645799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:10.645803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:10.645936 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:10.646374 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:10.646379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:10.646383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:10.646766 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:10.647052 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.647395 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.647657 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 19:34:10.652825 17559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:34:10.659793 17559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:34:10.699532 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.676804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.0.97.1:42227)
0923 19:34:10.689987 (+ 13183us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.690000 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.690034 (+ 34us) server_negotiation.cc:407] Connection header received
0923 19:34:10.690176 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.690198 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.690596 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.690863 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:10.691813 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.693244 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.696919 (+ 3675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.698108 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.698334 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.698618 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.698944 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.699150 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12930,"thread_start_us":12724,"threads_started":1}
I20260923 19:34:10.699555 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.675875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42227 (local address 127.0.0.1:50018)
0923 19:34:10.686885 (+ 11010us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:10.686945 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:10.687142 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:10.690904 (+ 3762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:10.690910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:10.690945 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:10.691550 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.691564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.693406 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.693413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.695504 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.695519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.698443 (+ 2924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.698449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.698683 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.698735 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 19:34:10.698876 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 19:34:10.699129 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10730,"thread_start_us":14074,"threads_started":1}
I20260923 19:34:10.700150 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.676434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.0.97.2:42355)
0923 19:34:10.677020 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.677031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.687532 (+ 10501us) server_negotiation.cc:407] Connection header received
0923 19:34:10.691311 (+ 3779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.691321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.691561 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.691728 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:10.692831 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.694251 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.696825 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.698346 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.698481 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.698697 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.699650 (+ 953us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.699823 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":832,"threads_started":1}
I20260923 19:34:10.701314 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.676899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42303 (local address 127.0.0.1:36660)
0923 19:34:10.690227 (+ 13328us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:10.690247 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:10.690398 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:10.693480 (+ 3082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:10.693486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:10.693517 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:10.693923 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.693931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.696278 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.696282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.698454 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.698467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.700581 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.700593 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.700763 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.700794 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 19:34:10.700858 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 19:34:10.700992 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13207,"thread_start_us":10687,"threads_started":1}
I20260923 19:34:10.701336 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.676171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42355 (local address 127.0.0.1:35194)
0923 19:34:10.687303 (+ 11132us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:10.687341 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:10.689196 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:10.692083 (+ 2887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:10.692089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:10.692125 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:10.692589 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.692602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.694414 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.694423 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.696664 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.696677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.698849 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.698856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.699049 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.699086 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 19:34:10.699176 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 19:34:10.699347 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10925,"thread_start_us":10276,"threads_started":1}
I20260923 19:34:10.702386 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.677555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.0.97.3:42303)
0923 19:34:10.692732 (+ 15177us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.692742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.692773 (+ 31us) server_negotiation.cc:407] Connection header received
0923 19:34:10.692858 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.692863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.693018 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.694085 (+ 1067us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:10.694351 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.696075 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.698679 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.700187 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.701569 (+ 1382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.701777 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.701856 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.702024 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14965,"thread_start_us":15604,"threads_started":1}
I20260923 19:34:10.705442 17683 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0ec714972e4a47ac1395e023dac908 (DEFAULT_TABLE table=test-table [id=728a3f56048944c59199970634494f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:10.705519 17817 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0ec714972e4a47ac1395e023dac908 (DEFAULT_TABLE table=test-table [id=728a3f56048944c59199970634494f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:10.706033 17951 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0ec714972e4a47ac1395e023dac908 (DEFAULT_TABLE table=test-table [id=728a3f56048944c59199970634494f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:10.707384 17683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0ec714972e4a47ac1395e023dac908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:10.707406 17951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0ec714972e4a47ac1395e023dac908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:10.707553 17817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0ec714972e4a47ac1395e023dac908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:10.720750 18034 tablet_bootstrap.cc:492] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: Bootstrap starting.
I20260923 19:34:10.725126 18034 tablet_bootstrap.cc:654] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:10.726444 18034 log.cc:824] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:10.728134 18035 tablet_bootstrap.cc:492] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: Bootstrap starting.
I20260923 19:34:10.729102 18036 tablet_bootstrap.cc:492] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: Bootstrap starting.
I20260923 19:34:10.730226 18034 tablet_bootstrap.cc:492] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: No bootstrap required, opened a new log
I20260923 19:34:10.730619 18034 ts_tablet_manager.cc:1397] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s
I20260923 19:34:10.734161 18035 tablet_bootstrap.cc:654] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:10.735786 18035 log.cc:824] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:10.736780 18036 tablet_bootstrap.cc:654] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:10.737957 18036 log.cc:824] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:10.739580 18035 tablet_bootstrap.cc:492] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: No bootstrap required, opened a new log
I20260923 19:34:10.739909 18035 ts_tablet_manager.cc:1397] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260923 19:34:10.740661 18036 tablet_bootstrap.cc:492] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: No bootstrap required, opened a new log
I20260923 19:34:10.740918 18036 ts_tablet_manager.cc:1397] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260923 19:34:10.741151 18034 raft_consensus.cc:348] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } }
I20260923 19:34:10.741691 18034 raft_consensus.cc:374] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:10.741845 18034 raft_consensus.cc:729] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd33d746d89643d79cd78692a9dd6847, State: Initialized, Role: FOLLOWER
I20260923 19:34:10.742640 18034 consensus_queue.cc:260] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } }
I20260923 19:34:10.746277 18034 ts_tablet_manager.cc:1428] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260923 19:34:10.749521 18036 raft_consensus.cc:348] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } }
I20260923 19:34:10.749574 18035 raft_consensus.cc:348] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } }
I20260923 19:34:10.749961 18036 raft_consensus.cc:374] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:10.750028 18035 raft_consensus.cc:374] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:10.750098 18036 raft_consensus.cc:729] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089749e01eb44708a994a5447523cbcd, State: Initialized, Role: FOLLOWER
I20260923 19:34:10.750140 18035 raft_consensus.cc:729] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade0569071d840a8be88370770a142ea, State: Initialized, Role: FOLLOWER
I20260923 19:34:10.750720 18036 consensus_queue.cc:260] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } }
I20260923 19:34:10.750952 18035 consensus_queue.cc:260] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } }
I20260923 19:34:10.751789 18016 heartbeater.cc:503] Master 127.0.97.62:33455 was elected leader, sending a full tablet report...
I20260923 19:34:10.752401 18036 ts_tablet_manager.cc:1428] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260923 19:34:10.753203 18040 raft_consensus.cc:482] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:34:10.753471 18040 raft_consensus.cc:504] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } }
I20260923 19:34:10.755407 18040 leader_election.cc:288] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ade0569071d840a8be88370770a142ea (127.0.97.2:42355), 089749e01eb44708a994a5447523cbcd (127.0.97.3:42303)
I20260923 19:34:10.758410 17882 heartbeater.cc:503] Master 127.0.97.62:33455 was elected leader, sending a full tablet report...
I20260923 19:34:10.759791 18035 ts_tablet_manager.cc:1428] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: Time spent starting tablet: real 0.020s user 0.003s sys 0.016s
I20260923 19:34:10.765530 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.755649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42355 (local address 127.0.97.1:55069)
0923 19:34:10.758785 (+ 3136us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:10.758810 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:10.758976 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:10.759511 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:10.759518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:10.759556 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:10.759982 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.759996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.761147 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.761151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.763215 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.763228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.764720 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.764726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.764864 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.764890 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 19:34:10.764997 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 19:34:10.765134 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2981,"mutex_wait_us":3,"thread_start_us":2754,"threads_started":1}
I20260923 19:34:10.765514 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.756100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55069 (local address 127.0.97.2:42355)
0923 19:34:10.756309 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.756316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.758929 (+ 2613us) server_negotiation.cc:407] Connection header received
0923 19:34:10.759130 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.759135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.759212 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.759317 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:10.760144 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.760897 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.763354 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.764530 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.764640 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.764732 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.765036 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.765222 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 19:34:10.766741 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.755760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42303 (local address 127.0.97.1:53989)
0923 19:34:10.759495 (+ 3735us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:10.759514 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:10.759731 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:10.760353 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:10.760357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:10.760383 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:10.760779 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.760791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.762201 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.762206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.764237 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:10.764252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.765994 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.766001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:10.766172 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.766201 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 19:34:10.766279 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 19:34:10.766434 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3612,"mutex_wait_us":30,"thread_start_us":4429,"threads_started":1}
I20260923 19:34:10.766793 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.756306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53989 (local address 127.0.97.3:42303)
0923 19:34:10.756516 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.756523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.759601 (+ 3078us) server_negotiation.cc:407] Connection header received
0923 19:34:10.759901 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.759907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.759999 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.760063 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:10.760988 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.762047 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.764392 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.765590 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.765740 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.765836 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.766379 (+ 543us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.766560 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 19:34:10.766587 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0ec714972e4a47ac1395e023dac908" candidate_uuid: "dd33d746d89643d79cd78692a9dd6847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade0569071d840a8be88370770a142ea" is_pre_election: true
I20260923 19:34:10.767146 17837 raft_consensus.cc:2454] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd33d746d89643d79cd78692a9dd6847 in term 0.
I20260923 19:34:10.767807 17637 leader_election.cc:302] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ade0569071d840a8be88370770a142ea, dd33d746d89643d79cd78692a9dd6847; no voters:
I20260923 19:34:10.767864 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0ec714972e4a47ac1395e023dac908" candidate_uuid: "dd33d746d89643d79cd78692a9dd6847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089749e01eb44708a994a5447523cbcd" is_pre_election: true
I20260923 19:34:10.768193 18040 raft_consensus.cc:2781] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:34:10.768316 18040 raft_consensus.cc:482] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:34:10.768409 18040 raft_consensus.cc:3037] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:10.768432 17971 raft_consensus.cc:2454] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd33d746d89643d79cd78692a9dd6847 in term 0.
I20260923 19:34:10.770879 18040 raft_consensus.cc:504] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } }
I20260923 19:34:10.771493 18040 leader_election.cc:288] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [CANDIDATE]: Term 1 election: Requested vote from peers ade0569071d840a8be88370770a142ea (127.0.97.2:42355), 089749e01eb44708a994a5447523cbcd (127.0.97.3:42303)
I20260923 19:34:10.771927 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0ec714972e4a47ac1395e023dac908" candidate_uuid: "dd33d746d89643d79cd78692a9dd6847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade0569071d840a8be88370770a142ea"
I20260923 19:34:10.772131 17837 raft_consensus.cc:3037] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:10.772112 17971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0ec714972e4a47ac1395e023dac908" candidate_uuid: "dd33d746d89643d79cd78692a9dd6847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089749e01eb44708a994a5447523cbcd"
I20260923 19:34:10.772349 17971 raft_consensus.cc:3037] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:10.775214 17837 raft_consensus.cc:2454] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd33d746d89643d79cd78692a9dd6847 in term 1.
I20260923 19:34:10.775382 17971 raft_consensus.cc:2454] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd33d746d89643d79cd78692a9dd6847 in term 1.
I20260923 19:34:10.775626 17637 leader_election.cc:302] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ade0569071d840a8be88370770a142ea, dd33d746d89643d79cd78692a9dd6847; no voters:
I20260923 19:34:10.775875 18040 raft_consensus.cc:2781] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:10.776331 18040 raft_consensus.cc:686] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 1 LEADER]: Becoming Leader. State: Replica: dd33d746d89643d79cd78692a9dd6847, State: Running, Role: LEADER
I20260923 19:34:10.776772 18040 consensus_queue.cc:237] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } }
W20260923 19:34:10.779043 17749 tablet.cc:2369] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:34:10.781158 17559 catalog_manager.cc:5495] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd33d746d89643d79cd78692a9dd6847 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "dd33d746d89643d79cd78692a9dd6847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd33d746d89643d79cd78692a9dd6847" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 } health_report { overall_health: UNKNOWN } } }
I20260923 19:34:10.797677 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.791873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.0.97.1:42227)
0923 19:34:10.792090 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.792097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.792165 (+ 68us) server_negotiation.cc:407] Connection header received
0923 19:34:10.792438 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.792443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.792531 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.792673 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:34:10.793074 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.793839 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.795386 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.795804 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.795913 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.796401 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:34:10.797093 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:34:10.797229 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.797307 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.797395 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 19:34:10.806653 17971 raft_consensus.cc:1261] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [term 1 FOLLOWER]: Refusing update from remote peer dd33d746d89643d79cd78692a9dd6847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:10.806703 17837 raft_consensus.cc:1261] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [term 1 FOLLOWER]: Refusing update from remote peer dd33d746d89643d79cd78692a9dd6847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:10.807619 18045 consensus_queue.cc:1035] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "089749e01eb44708a994a5447523cbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:10.808063 18040 consensus_queue.cc:1035] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade0569071d840a8be88370770a142ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:10.820534 18048 mvcc.cc:205] Tried to move back new op lower bound from 7332626640092364800 to 7332626639985180672. Current Snapshot: MvccSnapshot[applied={T|T < 7332626640092364800}]
I20260923 19:34:10.835523 17558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:10.835883 17558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:10.840001 17558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c7fa66b211649a28ae1214808f746fc: Sending DeleteTablet for 3 replicas of tablet 6a0ec714972e4a47ac1395e023dac908
I20260923 19:34:10.841380 17817 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a0ec714972e4a47ac1395e023dac908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35194
I20260923 19:34:10.841615 17951 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a0ec714972e4a47ac1395e023dac908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36660
I20260923 19:34:10.841629 17683 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a0ec714972e4a47ac1395e023dac908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50018
I20260923 19:34:10.843765 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 17620
I20260923 19:34:10.854871 18059 tablet_replica.cc:330] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: stopping tablet replica
I20260923 19:34:10.855710 18059 raft_consensus.cc:2229] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:34:10.857045 18059 raft_consensus.cc:2258] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:10.858160 18060 tablet_replica.cc:330] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: stopping tablet replica
I20260923 19:34:10.858743 18060 raft_consensus.cc:2229] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:10.861310 18059 ts_tablet_manager.cc:1905] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:10.862402 18060 raft_consensus.cc:2258] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:10.862525 18058 tablet_replica.cc:330] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: stopping tablet replica
I20260923 19:34:10.865432 18060 ts_tablet_manager.cc:1905] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:10.866344 18058 raft_consensus.cc:2229] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:10.866869 18058 raft_consensus.cc:2258] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:10.872846 18059 ts_tablet_manager.cc:1918] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:10.873131 18059 log.cc:1196] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/6a0ec714972e4a47ac1395e023dac908
I20260923 19:34:10.873817 18059 ts_tablet_manager.cc:1939] T 6a0ec714972e4a47ac1395e023dac908 P dd33d746d89643d79cd78692a9dd6847: Deleting consensus metadata
I20260923 19:34:10.876157 18060 ts_tablet_manager.cc:1918] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:10.876206 17546 catalog_manager.cc:4841] TS dd33d746d89643d79cd78692a9dd6847 (127.0.97.1:42227): tablet 6a0ec714972e4a47ac1395e023dac908 (table test-table [id=728a3f56048944c59199970634494f95]) successfully deleted
I20260923 19:34:10.876350 18060 log.cc:1196] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/6a0ec714972e4a47ac1395e023dac908
I20260923 19:34:10.876847 18060 ts_tablet_manager.cc:1939] T 6a0ec714972e4a47ac1395e023dac908 P 089749e01eb44708a994a5447523cbcd: Deleting consensus metadata
I20260923 19:34:10.878525 17543 catalog_manager.cc:4841] TS 089749e01eb44708a994a5447523cbcd (127.0.97.3:42303): tablet 6a0ec714972e4a47ac1395e023dac908 (table test-table [id=728a3f56048944c59199970634494f95]) successfully deleted
I20260923 19:34:10.880316 18058 ts_tablet_manager.cc:1905] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 19:34:10 dist-test-slave-kxfb krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:34:10.886823 18058 ts_tablet_manager.cc:1918] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:10.886982 18058 log.cc:1196] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/6a0ec714972e4a47ac1395e023dac908
I20260923 19:34:10.887578 18058 ts_tablet_manager.cc:1939] T 6a0ec714972e4a47ac1395e023dac908 P ade0569071d840a8be88370770a142ea: Deleting consensus metadata
I20260923 19:34:10.889266 17544 catalog_manager.cc:4841] TS ade0569071d840a8be88370770a142ea (127.0.97.2:42355): tablet 6a0ec714972e4a47ac1395e023dac908 (table test-table [id=728a3f56048944c59199970634494f95]) successfully deleted
I20260923 19:34:10.891554 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:10.858347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.0.97.1:42227)
0923 19:34:10.858524 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:10.858531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:10.859622 (+ 1091us) server_negotiation.cc:407] Connection header received
0923 19:34:10.859938 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:10.859944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:10.860044 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:10.860128 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:10.862735 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.863998 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.867225 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:10.867965 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:10.868065 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:10.883415 (+ 15350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:10.883435 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:10.883453 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:10.883524 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:10.887826 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:10.888915 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:10.888923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:10.888926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:10.889102 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:10.889647 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:10.889654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:10.889658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:10.890231 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:10.890646 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:10.890844 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 19:34:10.891148 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 19:34:10.892360 17743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:10.996773 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 17620
I20260923 19:34:11.016255 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 17754
Sep 23 19:34:11 dist-test-slave-kxfb krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:34:11.031930 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:11.017080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.0.97.2:42355)
0923 19:34:11.017309 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:11.017314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:11.017335 (+ 21us) server_negotiation.cc:407] Connection header received
0923 19:34:11.017409 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:11.017414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:11.017500 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:11.017596 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:11.018880 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:11.019707 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:11.020947 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:11.021333 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:11.021430 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:11.025767 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:11.025783 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:11.025802 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:11.025877 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:11.029041 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:11.029865 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:11.029874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:11.029876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:11.030017 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:11.030459 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:11.030463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:11.030465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:11.030877 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:11.031143 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:11.031425 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 19:34:11.031621 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 19:34:11.032756 17877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:11.128527 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 17754
I20260923 19:34:11.143405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 17888
Sep 23 19:34:11 dist-test-slave-kxfb krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:34:11.159601 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:11.144302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.0.97.3:42303)
0923 19:34:11.144484 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:11.144489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:11.144636 (+ 147us) server_negotiation.cc:407] Connection header received
0923 19:34:11.144918 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:11.144923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:11.145003 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:11.145119 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:11.146818 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:11.147613 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:11.149123 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:11.149538 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:11.149633 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:11.154144 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:11.154158 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:11.154176 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:11.154218 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:11.156977 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:11.157682 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:11.157687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:11.157690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:11.157804 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:11.158102 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:11.158105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:11.158107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:11.158522 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:11.158731 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:11.159179 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 19:34:11.159344 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 19:34:11.160506 18011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:11.250885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 17888
I20260923 19:34:11.265033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 17527
I20260923 19:34:11.266392 17589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:11.377694 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 17527
2026-09-23T19:34:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4322 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T19:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:11Z Disabled control of system clock
I20260923 19:34:11.466563 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42669
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 19:34:11.833351 18069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:11.833653 18069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:11.833734 18069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:11.840972 18069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:11.841092 18069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:11.841130 18069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:11.841183 18069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:11.841218 18069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:11.850548 18069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:11.851810 18069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:11.853562 18069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:11.865355 18077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:11.865428 18074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:11.865428 18075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:11.866519 18069 server_base.cc:1034] running on GCE node
I20260923 19:34:11.867486 18069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:11.869174 18069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:11.870434 18069 hybrid_clock.cc:648] HybridClock initialized: now 1790192051870364 us; error 43 us; skew 500 ppm
I20260923 19:34:11.870865 18069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:11.879105 18069 webserver.cc:460] Webserver started at http://127.0.97.62:38035/ using document root <none> and password file <none>
I20260923 19:34:11.879983 18069 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:11.880098 18069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:11.880453 18069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:11.884127 18069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "367de5783448443fba31d3023710747e"
format_stamp: "Formatted at 2026-09-23 19:34:11 on dist-test-slave-kxfb"
server_key: "cf40026ce50c030749cc2021c0368269"
server_key_iv: "72dc3f2233311af40e34e427b3790910"
server_key_version: "encryptionkey@0"
I20260923 19:34:11.885028 18069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "367de5783448443fba31d3023710747e"
format_stamp: "Formatted at 2026-09-23 19:34:11 on dist-test-slave-kxfb"
server_key: "cf40026ce50c030749cc2021c0368269"
server_key_iv: "72dc3f2233311af40e34e427b3790910"
server_key_version: "encryptionkey@0"
I20260923 19:34:11.891568 18069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 19:34:11.896103 18083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:11.897957 18069 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:34:11.898169 18069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "367de5783448443fba31d3023710747e"
format_stamp: "Formatted at 2026-09-23 19:34:11 on dist-test-slave-kxfb"
server_key: "cf40026ce50c030749cc2021c0368269"
server_key_iv: "72dc3f2233311af40e34e427b3790910"
server_key_version: "encryptionkey@0"
I20260923 19:34:11.898461 18069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:11.912797 18069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:11.913836 18069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:11.914143 18069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:11.934881 18069 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42669
I20260923 19:34:11.934929 18134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42669 every 8 connection(s)
I20260923 19:34:11.936453 18069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:11.940546 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:11.944689 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 18069
I20260923 19:34:11.944876 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:11.945372 388 external_mini_cluster.cc:1428] Setting key e56a2846cf26292d63e60a0bea1ca843
I20260923 19:34:11.951234 18135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e: Bootstrap starting.
I20260923 19:34:11.954658 18135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:11.955878 18135 log.cc:824] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:11.959489 18135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e: No bootstrap required, opened a new log
I20260923 19:34:11.961665 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:11.947681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.0.97.62:42669)
0923 19:34:11.948921 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:11.948944 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:11.948997 (+ 53us) server_negotiation.cc:407] Connection header received
0923 19:34:11.951021 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:11.951049 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:11.951643 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:11.952032 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:11.952724 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:11.954163 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:11.956184 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:11.957397 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:11.957605 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:11.958889 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:11.958897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:11.958915 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:11.959586 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:11.959706 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:11.959733 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:11.959845 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:11.960098 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 19:34:11.960507 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":434,"threads_started":1}
I20260923 19:34:11.967175 18135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367de5783448443fba31d3023710747e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42669 } }
I20260923 19:34:11.967517 18135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:11.967599 18135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367de5783448443fba31d3023710747e, State: Initialized, Role: FOLLOWER
I20260923 19:34:11.968137 18135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367de5783448443fba31d3023710747e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42669 } }
I20260923 19:34:11.968351 18135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:11.968478 18135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:11.968632 18135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:11.970840 18135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367de5783448443fba31d3023710747e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42669 } }
I20260923 19:34:11.971303 18135 leader_election.cc:302] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 367de5783448443fba31d3023710747e; no voters:
I20260923 19:34:11.971894 18135 leader_election.cc:288] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:11.972100 18140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:11.973030 18140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [term 1 LEADER]: Becoming Leader. State: Replica: 367de5783448443fba31d3023710747e, State: Running, Role: LEADER
I20260923 19:34:11.973875 18135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:11.973750 18140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367de5783448443fba31d3023710747e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42669 } }
I20260923 19:34:11.977192 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 367de5783448443fba31d3023710747e. Latest consensus state: current_term: 1 leader_uuid: "367de5783448443fba31d3023710747e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367de5783448443fba31d3023710747e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42669 } } }
I20260923 19:34:11.977504 18141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "367de5783448443fba31d3023710747e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367de5783448443fba31d3023710747e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42669 } } }
I20260923 19:34:11.978147 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:11.978143 18141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e [sys.catalog]: This master's current role is: LEADER
W20260923 19:34:11.988793 18156 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:34:11.988953 18156 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:34:11.989116 18150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:11.991495 18150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:12.001845 18150 catalog_manager.cc:1347] Generated new cluster ID: 756f1fc81f9a48f8b43c700e0aa4419f
I20260923 19:34:12.001989 18150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:12.021669 18150 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:12.023535 18150 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:12.031648 18150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e: Generated new TSK 0
I20260923 19:34:12.032836 18150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:12.040498 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42669
--builtin_ntp_servers=127.0.97.20:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 19:34:12.420356 18159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:12.420707 18159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:12.420799 18159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:12.428758 18159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:12.428892 18159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:12.429013 18159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:34:12.439239 18159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:12.440851 18159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:12.442939 18159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:12.457661 18164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:12.457824 18165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:12.458226 18167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:12.459007 18159 server_base.cc:1034] running on GCE node
I20260923 19:34:12.459705 18159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:12.460870 18159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:12.462100 18159 hybrid_clock.cc:648] HybridClock initialized: now 1790192052462030 us; error 63 us; skew 500 ppm
I20260923 19:34:12.462520 18159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:12.465099 18159 webserver.cc:460] Webserver started at http://127.0.97.1:38587/ using document root <none> and password file <none>
I20260923 19:34:12.465888 18159 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:12.465986 18159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:12.466382 18159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:12.469647 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "082ae47383e84f268507c797fa1c8475"
format_stamp: "Formatted at 2026-09-23 19:34:12 on dist-test-slave-kxfb"
server_key: "2e9b287ec75b2332712ab8624c961856"
server_key_iv: "26886f695f63d3d090b49e320c843d24"
server_key_version: "encryptionkey@0"
I20260923 19:34:12.470510 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "082ae47383e84f268507c797fa1c8475"
format_stamp: "Formatted at 2026-09-23 19:34:12 on dist-test-slave-kxfb"
server_key: "2e9b287ec75b2332712ab8624c961856"
server_key_iv: "26886f695f63d3d090b49e320c843d24"
server_key_version: "encryptionkey@0"
I20260923 19:34:12.476491 18159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 19:34:12.480890 18173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:12.482630 18159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:34:12.482836 18159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "082ae47383e84f268507c797fa1c8475"
format_stamp: "Formatted at 2026-09-23 19:34:12 on dist-test-slave-kxfb"
server_key: "2e9b287ec75b2332712ab8624c961856"
server_key_iv: "26886f695f63d3d090b49e320c843d24"
server_key_version: "encryptionkey@0"
I20260923 19:34:12.483078 18159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:12.499595 18159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:12.500427 18159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:12.500679 18159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:12.501686 18159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:12.503351 18159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:12.503466 18159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:12.503567 18159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:12.503636 18159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:12.536970 18159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44403
I20260923 19:34:12.537056 18285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44403 every 8 connection(s)
I20260923 19:34:12.538390 18159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:34:12.539023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 18159
I20260923 19:34:12.539227 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:34:12.539841 388 external_mini_cluster.cc:1428] Setting key 04b10254ed7109185b00924866bc327c
I20260923 19:34:12.543553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42669
--builtin_ntp_servers=127.0.97.20:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 19:34:12.552126 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:12.541555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40033 (local address 127.0.97.62:42669)
0923 19:34:12.541985 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:12.541991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:12.543783 (+ 1792us) server_negotiation.cc:407] Connection header received
0923 19:34:12.545814 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:12.545820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:12.545905 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:12.545999 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:12.547168 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:12.547890 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:12.549422 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:12.549820 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:12.549931 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:12.551273 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:12.551277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:12.551280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:12.551298 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:12.551341 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:12.551353 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:12.551437 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:12.551660 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 19:34:12.551756 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":225,"threads_started":1}
I20260923 19:34:12.553095 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:12.542416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42669 (local address 127.0.97.1:40033)
0923 19:34:12.543606 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:12.543657 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:12.545537 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:12.546275 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:12.546284 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:12.546418 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:12.547003 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:12.547021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:12.548047 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:12.548051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:12.549269 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:12.549282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:12.550056 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:12.550064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:12.550278 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:12.550968 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:34:12.550971 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:12.551010 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:34:12.551091 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:12.551450 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:12.551462 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:12.551470 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 19:34:12.551616 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 19:34:12.552017 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":413,"threads_started":1}
I20260923 19:34:12.554855 18286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42669
I20260923 19:34:12.555292 18286 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:12.556417 18286 heartbeater.cc:511] Master 127.0.97.62:42669 requested a full tablet report, sending...
I20260923 19:34:12.559168 18100 ts_manager.cc:194] Registered new tserver with Master: 082ae47383e84f268507c797fa1c8475 (127.0.97.1:44403)
I20260923 19:34:12.561281 18100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:40033
W20260923 19:34:12.900629 18290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:12.900913 18290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:12.900995 18290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:12.908182 18290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:12.908273 18290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:12.908375 18290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:34:12.917186 18290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:12.918418 18290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:12.920670 18290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:12.937327 18295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:12.937366 18296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:12.939402 18290 server_base.cc:1034] running on GCE node
W20260923 19:34:12.945245 18298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:12.946190 18290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:12.947660 18290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:12.948908 18290 hybrid_clock.cc:648] HybridClock initialized: now 1790192052948814 us; error 87 us; skew 500 ppm
I20260923 19:34:12.949273 18290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:12.952334 18290 webserver.cc:460] Webserver started at http://127.0.97.2:34499/ using document root <none> and password file <none>
I20260923 19:34:12.953294 18290 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:12.953418 18290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:12.953817 18290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:12.958037 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "adabd47e19444b5a9b4d609892cba6be"
format_stamp: "Formatted at 2026-09-23 19:34:12 on dist-test-slave-kxfb"
server_key: "a8d81cc6dfd1bce3797e6079b08db1e1"
server_key_iv: "b3482319289b80b2940e03a694d789f0"
server_key_version: "encryptionkey@0"
I20260923 19:34:12.959098 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "adabd47e19444b5a9b4d609892cba6be"
format_stamp: "Formatted at 2026-09-23 19:34:12 on dist-test-slave-kxfb"
server_key: "a8d81cc6dfd1bce3797e6079b08db1e1"
server_key_iv: "b3482319289b80b2940e03a694d789f0"
server_key_version: "encryptionkey@0"
I20260923 19:34:12.966570 18290 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 19:34:12.971808 18304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:12.973739 18290 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260923 19:34:12.973939 18290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "adabd47e19444b5a9b4d609892cba6be"
format_stamp: "Formatted at 2026-09-23 19:34:12 on dist-test-slave-kxfb"
server_key: "a8d81cc6dfd1bce3797e6079b08db1e1"
server_key_iv: "b3482319289b80b2940e03a694d789f0"
server_key_version: "encryptionkey@0"
I20260923 19:34:12.974207 18290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:12.995814 18290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:12.996634 18290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:12.996868 18290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:12.997889 18290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:12.999464 18290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:12.999562 18290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:12.999675 18290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:12.999742 18290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:13.033353 18290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43267
I20260923 19:34:13.033454 18416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43267 every 8 connection(s)
I20260923 19:34:13.034731 18290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:34:13.035271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 18290
I20260923 19:34:13.035467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:34:13.036116 388 external_mini_cluster.cc:1428] Setting key 82f236ecf5fb96c953544a539aa79bcb
I20260923 19:34:13.039852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42669
--builtin_ntp_servers=127.0.97.20:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 19:34:13.049619 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.038048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52545 (local address 127.0.97.62:42669)
0923 19:34:13.038319 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.038333 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.040217 (+ 1884us) server_negotiation.cc:407] Connection header received
0923 19:34:13.042362 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.042369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.042464 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.042559 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:13.043885 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.044678 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.046407 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.046982 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.047118 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.048748 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:13.048754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:13.048760 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:13.048786 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:13.048843 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:13.048859 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:13.048958 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.049159 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.049275 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 19:34:13.050663 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.038717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42669 (local address 127.0.97.2:52545)
0923 19:34:13.039943 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:13.040046 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:13.042007 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:13.042863 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:13.042873 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:13.042986 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:13.043692 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.043710 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.044848 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.044852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.046239 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.046253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.047295 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.047303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.047585 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.048356 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:34:13.048359 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:13.048403 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:34:13.048506 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:13.048958 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:13.048965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:13.048970 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 19:34:13.049131 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 19:34:13.049484 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":414,"threads_started":1}
I20260923 19:34:13.052268 18417 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42669
I20260923 19:34:13.052673 18417 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:13.053759 18417 heartbeater.cc:511] Master 127.0.97.62:42669 requested a full tablet report, sending...
I20260923 19:34:13.055603 18100 ts_manager.cc:194] Registered new tserver with Master: adabd47e19444b5a9b4d609892cba6be (127.0.97.2:43267)
I20260923 19:34:13.056479 18100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:52545
W20260923 19:34:13.420408 18420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:13.420763 18420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:13.420885 18420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:13.428853 18420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:13.428987 18420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:13.429157 18420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:34:13.439164 18420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:13.440819 18420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:13.442955 18420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:13.458314 18425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:13.458400 18426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:13.458585 18428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:13.459301 18420 server_base.cc:1034] running on GCE node
I20260923 19:34:13.460057 18420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:13.461206 18420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:13.462402 18420 hybrid_clock.cc:648] HybridClock initialized: now 1790192053462327 us; error 48 us; skew 500 ppm
I20260923 19:34:13.462879 18420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:13.466259 18420 webserver.cc:460] Webserver started at http://127.0.97.3:37317/ using document root <none> and password file <none>
I20260923 19:34:13.467415 18420 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:13.467562 18420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:13.467994 18420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:13.471926 18420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa"
format_stamp: "Formatted at 2026-09-23 19:34:13 on dist-test-slave-kxfb"
server_key: "22132e1d313c09cb7b4b86fb6c6c78e1"
server_key_iv: "d1c00e12727411bcd85d002813db275b"
server_key_version: "encryptionkey@0"
I20260923 19:34:13.472854 18420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa"
format_stamp: "Formatted at 2026-09-23 19:34:13 on dist-test-slave-kxfb"
server_key: "22132e1d313c09cb7b4b86fb6c6c78e1"
server_key_iv: "d1c00e12727411bcd85d002813db275b"
server_key_version: "encryptionkey@0"
I20260923 19:34:13.479516 18420 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 19:34:13.483920 18434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:13.485605 18420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 19:34:13.485800 18420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa"
format_stamp: "Formatted at 2026-09-23 19:34:13 on dist-test-slave-kxfb"
server_key: "22132e1d313c09cb7b4b86fb6c6c78e1"
server_key_iv: "d1c00e12727411bcd85d002813db275b"
server_key_version: "encryptionkey@0"
I20260923 19:34:13.486078 18420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:13.510428 18420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:13.511394 18420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:13.511667 18420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:13.512753 18420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:13.514465 18420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:13.514557 18420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:13.514670 18420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:13.514744 18420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:13.550319 18420 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39641
I20260923 19:34:13.550477 18546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39641 every 8 connection(s)
I20260923 19:34:13.551882 18420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:34:13.560052 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 18420
I20260923 19:34:13.560235 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:34:13.560768 388 external_mini_cluster.cc:1428] Setting key 083904371b1623e15161acd1464652cb
I20260923 19:34:13.565347 18286 heartbeater.cc:503] Master 127.0.97.62:42669 was elected leader, sending a full tablet report...
I20260923 19:34:13.568305 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.555457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45039 (local address 127.0.97.62:42669)
0923 19:34:13.556040 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.556045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.557500 (+ 1455us) server_negotiation.cc:407] Connection header received
0923 19:34:13.559629 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.559636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.559746 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.559846 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:13.561282 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.562453 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.564917 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.565516 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.565665 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.567492 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:13.567497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:13.567501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:13.567522 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:13.567569 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:13.567582 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:13.567661 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.567889 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.567991 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":284,"threads_started":1}
I20260923 19:34:13.569721 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.556114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42669 (local address 127.0.97.3:45039)
0923 19:34:13.557304 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:13.557350 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:13.559296 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:13.560117 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:13.560126 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:13.560263 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:13.561052 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.561079 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.562669 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.562677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.564707 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.564733 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.565792 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.565801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.566162 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.567038 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:34:13.567042 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:13.567114 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:34:13.567286 (+ 172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:13.567668 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:13.567675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:13.567681 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 19:34:13.567860 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 19:34:13.568335 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":357,"threads_started":1}
I20260923 19:34:13.571656 18547 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42669
I20260923 19:34:13.572064 18547 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:13.573045 18547 heartbeater.cc:511] Master 127.0.97.62:42669 requested a full tablet report, sending...
I20260923 19:34:13.575017 18100 ts_manager.cc:194] Registered new tserver with Master: e72d0b4f14944e7ab46bb3d3e7df6bfa (127.0.97.3:39641)
I20260923 19:34:13.576116 18100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:45039
I20260923 19:34:13.577327 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:34:13.589287 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.582934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.0.97.62:42669)
0923 19:34:13.583194 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.583201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.583611 (+ 410us) server_negotiation.cc:407] Connection header received
0923 19:34:13.583824 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.583828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.583912 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.583998 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:13.584526 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.585231 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.586807 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.587219 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.587333 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.588559 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:13.588564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:13.588567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:13.588588 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:13.588622 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:13.588635 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:13.588712 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.588843 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.588985 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 19:34:13.594641 18100 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:34:13.602264 18100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:34:13.652148 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.618609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.0.97.2:43267)
0923 19:34:13.641504 (+ 22895us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.641513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.642367 (+ 854us) server_negotiation.cc:407] Connection header received
0923 19:34:13.642853 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.642863 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.643152 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.643341 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:13.644476 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.645867 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.649003 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.650723 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.650896 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.651251 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.651501 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.651725 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22736,"thread_start_us":22625,"threads_started":1}
I20260923 19:34:13.653025 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.617468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43267 (local address 127.0.0.1:44746)
0923 19:34:13.642135 (+ 24667us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:13.642197 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:13.642563 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:13.643665 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:13.643676 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:13.643716 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:13.644247 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.644261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.646062 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.646068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.648687 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.648709 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.650955 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.650963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.651192 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.651257 (+ 65us) client_negotiation.cc:769] Sending connection context
0923 19:34:13.651432 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 19:34:13.652672 (+ 1240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24273,"thread_start_us":26807,"threads_started":1}
I20260923 19:34:13.654102 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.617560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39641 (local address 127.0.0.1:43310)
0923 19:34:13.643887 (+ 26327us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:13.643911 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:13.644017 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:13.645906 (+ 1889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:13.645911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:13.645939 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:13.646600 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.646618 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.648271 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.648275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.650906 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.650930 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.653084 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.653091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.653411 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.653450 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 19:34:13.653531 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 19:34:13.653706 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26163,"mutex_wait_us":326,"thread_start_us":26010,"threads_started":1}
I20260923 19:34:13.654438 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.618376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.0.97.3:39641)
0923 19:34:13.645102 (+ 26726us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.645113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.645156 (+ 43us) server_negotiation.cc:407] Connection header received
0923 19:34:13.645278 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.645287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.645620 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.645889 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:13.646847 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.648102 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.651193 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.652843 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.652976 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.653387 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.653619 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.653832 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26526,"thread_start_us":26333,"threads_started":1}
I20260923 19:34:13.655205 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.618679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.0.97.1:44403)
0923 19:34:13.640871 (+ 22192us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.640882 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.644889 (+ 4007us) server_negotiation.cc:407] Connection header received
0923 19:34:13.645287 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.645296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.645620 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.645889 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:13.647202 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.648760 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.651556 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.653008 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.653158 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.653445 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.654680 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.654897 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21951,"thread_start_us":22214,"threads_started":1}
I20260923 19:34:13.655870 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.617828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44403 (local address 127.0.0.1:59798)
0923 19:34:13.644666 (+ 26838us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:13.644691 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:13.645018 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:13.646268 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:13.646272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:13.646340 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:13.646923 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.646944 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.648999 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.649004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.651312 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.651325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.653579 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.653587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.654096 (+ 509us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.654132 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 19:34:13.654215 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 19:34:13.655534 (+ 1319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26636,"mutex_wait_us":62,"thread_start_us":24972,"threads_started":1}
I20260923 19:34:13.657200 18352 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad4d3b3a025447e98f12200e529b1ca (DEFAULT_TABLE table=test-table [id=3c0d3dd5079f4da4a7494760149fe3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:13.658913 18352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad4d3b3a025447e98f12200e529b1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:13.658751 18482 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad4d3b3a025447e98f12200e529b1ca (DEFAULT_TABLE table=test-table [id=3c0d3dd5079f4da4a7494760149fe3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:13.660384 18482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad4d3b3a025447e98f12200e529b1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:13.660035 18221 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad4d3b3a025447e98f12200e529b1ca (DEFAULT_TABLE table=test-table [id=3c0d3dd5079f4da4a7494760149fe3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:13.661499 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad4d3b3a025447e98f12200e529b1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:13.672561 18566 tablet_bootstrap.cc:492] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: Bootstrap starting.
I20260923 19:34:13.675237 18567 tablet_bootstrap.cc:492] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: Bootstrap starting.
I20260923 19:34:13.677182 18566 tablet_bootstrap.cc:654] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:13.677139 18568 tablet_bootstrap.cc:492] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475: Bootstrap starting.
I20260923 19:34:13.679194 18567 tablet_bootstrap.cc:654] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:13.679314 18566 log.cc:824] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:13.680382 18567 log.cc:824] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:13.680727 18568 tablet_bootstrap.cc:654] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:13.681881 18568 log.cc:824] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:13.683236 18566 tablet_bootstrap.cc:492] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: No bootstrap required, opened a new log
I20260923 19:34:13.683564 18566 ts_tablet_manager.cc:1397] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 19:34:13.684629 18568 tablet_bootstrap.cc:492] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475: No bootstrap required, opened a new log
I20260923 19:34:13.684811 18567 tablet_bootstrap.cc:492] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: No bootstrap required, opened a new log
I20260923 19:34:13.684908 18568 ts_tablet_manager.cc:1397] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 19:34:13.685093 18567 ts_tablet_manager.cc:1397] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 19:34:13.692492 18568 raft_consensus.cc:348] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } }
I20260923 19:34:13.692900 18568 raft_consensus.cc:374] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:13.692986 18568 raft_consensus.cc:729] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082ae47383e84f268507c797fa1c8475, State: Initialized, Role: FOLLOWER
I20260923 19:34:13.693621 18568 consensus_queue.cc:260] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } }
I20260923 19:34:13.695832 18567 raft_consensus.cc:348] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } }
I20260923 19:34:13.696369 18567 raft_consensus.cc:374] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:13.696487 18567 raft_consensus.cc:729] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adabd47e19444b5a9b4d609892cba6be, State: Initialized, Role: FOLLOWER
I20260923 19:34:13.696663 18568 ts_tablet_manager.cc:1428] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260923 19:34:13.697240 18567 consensus_queue.cc:260] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } }
I20260923 19:34:13.695832 18566 raft_consensus.cc:348] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } }
I20260923 19:34:13.697820 18566 raft_consensus.cc:374] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:13.697979 18566 raft_consensus.cc:729] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72d0b4f14944e7ab46bb3d3e7df6bfa, State: Initialized, Role: FOLLOWER
I20260923 19:34:13.698825 18417 heartbeater.cc:503] Master 127.0.97.62:42669 was elected leader, sending a full tablet report...
I20260923 19:34:13.698743 18566 consensus_queue.cc:260] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } }
I20260923 19:34:13.699363 18567 ts_tablet_manager.cc:1428] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260923 19:34:13.701707 18547 heartbeater.cc:503] Master 127.0.97.62:42669 was elected leader, sending a full tablet report...
I20260923 19:34:13.702425 18566 ts_tablet_manager.cc:1428] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: Time spent starting tablet: real 0.019s user 0.006s sys 0.011s
W20260923 19:34:13.787389 18418 tablet.cc:2369] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:34:13.790975 18287 tablet.cc:2369] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:34:13.804073 18548 tablet.cc:2369] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:34:13.810078 18572 raft_consensus.cc:482] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:34:13.810398 18572 raft_consensus.cc:504] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } }
I20260923 19:34:13.812136 18572 leader_election.cc:288] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72d0b4f14944e7ab46bb3d3e7df6bfa (127.0.97.3:39641), adabd47e19444b5a9b4d609892cba6be (127.0.97.2:43267)
I20260923 19:34:13.818895 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.812608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54463 (local address 127.0.97.2:43267)
0923 19:34:13.812851 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.812858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.813617 (+ 759us) server_negotiation.cc:407] Connection header received
0923 19:34:13.813824 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.813828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.813899 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.813979 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:13.814629 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.815326 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.817220 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.817965 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.818078 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.818205 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.818507 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.818639 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 19:34:13.818948 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.812550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43267 (local address 127.0.97.1:54463)
0923 19:34:13.813490 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:13.813512 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:13.813680 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:13.814026 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:13.814031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:13.814068 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:13.814485 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.814500 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.815459 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.815465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.817034 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.817047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.818175 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.818181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.818350 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.818376 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 19:34:13.818455 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 19:34:13.818592 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":545,"threads_started":1}
I20260923 19:34:13.819679 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.812749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43613 (local address 127.0.97.3:39641)
0923 19:34:13.813004 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.813012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.813860 (+ 848us) server_negotiation.cc:407] Connection header received
0923 19:34:13.814071 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.814075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.814153 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.814232 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:13.814882 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.815674 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.817541 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.818488 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.818612 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.818726 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.819262 (+ 536us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.819409 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 19:34:13.820036 18372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad4d3b3a025447e98f12200e529b1ca" candidate_uuid: "082ae47383e84f268507c797fa1c8475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adabd47e19444b5a9b4d609892cba6be" is_pre_election: true
I20260923 19:34:13.820361 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.812758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39641 (local address 127.0.97.1:43613)
0923 19:34:13.813733 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:13.813750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:13.813922 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:13.814272 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:13.814276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:13.814329 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:13.814702 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.814712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.815821 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.815825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.817357 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:13.817370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.818867 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.818873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:13.819054 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.819088 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 19:34:13.819900 (+ 812us) client_negotiation.cc:240] Negotiation successful
0923 19:34:13.820054 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":573,"threads_started":1}
I20260923 19:34:13.820596 18372 raft_consensus.cc:2454] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082ae47383e84f268507c797fa1c8475 in term 0.
I20260923 19:34:13.821241 18177 leader_election.cc:302] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082ae47383e84f268507c797fa1c8475, adabd47e19444b5a9b4d609892cba6be; no voters:
I20260923 19:34:13.821398 18502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad4d3b3a025447e98f12200e529b1ca" candidate_uuid: "082ae47383e84f268507c797fa1c8475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" is_pre_election: true
I20260923 19:34:13.821623 18572 raft_consensus.cc:2781] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:34:13.821745 18572 raft_consensus.cc:482] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:34:13.821841 18572 raft_consensus.cc:3037] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:13.821976 18502 raft_consensus.cc:2454] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082ae47383e84f268507c797fa1c8475 in term 0.
I20260923 19:34:13.824167 18572 raft_consensus.cc:504] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } }
I20260923 19:34:13.824793 18572 leader_election.cc:288] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [CANDIDATE]: Term 1 election: Requested vote from peers e72d0b4f14944e7ab46bb3d3e7df6bfa (127.0.97.3:39641), adabd47e19444b5a9b4d609892cba6be (127.0.97.2:43267)
I20260923 19:34:13.825459 18502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad4d3b3a025447e98f12200e529b1ca" candidate_uuid: "082ae47383e84f268507c797fa1c8475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa"
I20260923 19:34:13.825456 18372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad4d3b3a025447e98f12200e529b1ca" candidate_uuid: "082ae47383e84f268507c797fa1c8475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adabd47e19444b5a9b4d609892cba6be"
I20260923 19:34:13.825700 18502 raft_consensus.cc:3037] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:13.825726 18372 raft_consensus.cc:3037] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:13.828112 18502 raft_consensus.cc:2454] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082ae47383e84f268507c797fa1c8475 in term 1.
I20260923 19:34:13.828527 18372 raft_consensus.cc:2454] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082ae47383e84f268507c797fa1c8475 in term 1.
I20260923 19:34:13.828588 18176 leader_election.cc:302] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082ae47383e84f268507c797fa1c8475, e72d0b4f14944e7ab46bb3d3e7df6bfa; no voters:
I20260923 19:34:13.828979 18572 raft_consensus.cc:2781] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:13.829546 18572 raft_consensus.cc:686] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 1 LEADER]: Becoming Leader. State: Replica: 082ae47383e84f268507c797fa1c8475, State: Running, Role: LEADER
I20260923 19:34:13.830148 18572 consensus_queue.cc:237] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } }
I20260923 19:34:13.835522 18099 catalog_manager.cc:5495] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 reported cstate change: term changed from 0 to 1, leader changed from <none> to 082ae47383e84f268507c797fa1c8475 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "082ae47383e84f268507c797fa1c8475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "082ae47383e84f268507c797fa1c8475" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44403 } health_report { overall_health: HEALTHY } } }
I20260923 19:34:13.879953 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.873930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.0.97.1:44403)
0923 19:34:13.874133 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.874138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.874160 (+ 22us) server_negotiation.cc:407] Connection header received
0923 19:34:13.874393 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.874398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.874520 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.874664 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:34:13.874992 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.875797 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.877355 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.877782 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.877874 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.878408 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:34:13.879218 (+ 810us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:34:13.879393 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.879546 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.879643 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 19:34:13.889648 18502 raft_consensus.cc:1261] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [term 1 FOLLOWER]: Refusing update from remote peer 082ae47383e84f268507c797fa1c8475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:13.889729 18372 raft_consensus.cc:1261] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [term 1 FOLLOWER]: Refusing update from remote peer 082ae47383e84f268507c797fa1c8475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:13.890864 18572 consensus_queue.cc:1035] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72d0b4f14944e7ab46bb3d3e7df6bfa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:13.891251 18577 consensus_queue.cc:1035] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adabd47e19444b5a9b4d609892cba6be" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:13.905711 18582 mvcc.cc:205] Tried to move back new op lower bound from 7332626652720517120 to 7332626652493103104. Current Snapshot: MvccSnapshot[applied={T|T < 7332626652720517120}]
I20260923 19:34:13.921870 18099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:13.922212 18099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:13.925839 18099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 367de5783448443fba31d3023710747e: Sending DeleteTablet for 3 replicas of tablet 4ad4d3b3a025447e98f12200e529b1ca
I20260923 19:34:13.927230 18352 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ad4d3b3a025447e98f12200e529b1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:13 UTC) from {username='slave'} at 127.0.0.1:44746
I20260923 19:34:13.927230 18482 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ad4d3b3a025447e98f12200e529b1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:13 UTC) from {username='slave'} at 127.0.0.1:43310
I20260923 19:34:13.927714 18221 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ad4d3b3a025447e98f12200e529b1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:13 UTC) from {username='slave'} at 127.0.0.1:59798
I20260923 19:34:13.934175 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 18159
I20260923 19:34:13.954406 18592 tablet_replica.cc:330] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475: stopping tablet replica
I20260923 19:34:13.955044 18592 raft_consensus.cc:2229] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:34:13.955854 18592 raft_consensus.cc:2258] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:13.964728 18592 ts_tablet_manager.cc:1905] T 4ad4d3b3a025447e98f12200e529b1ca P 082ae47383e84f268507c797fa1c8475: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:13.967109 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:13.935483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.0.97.1:44403)
0923 19:34:13.935680 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:13.935686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:13.948380 (+ 12694us) server_negotiation.cc:407] Connection header received
0923 19:34:13.948941 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:13.948949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:13.949085 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:13.949180 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:13.949904 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.951380 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.956618 (+ 5238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:13.957444 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:13.957558 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:13.960477 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:13.960492 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:13.960514 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:13.961480 (+ 966us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:13.961585 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:13.961616 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:13.961707 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:13.966492 (+ 4785us) server_negotiation.cc:299] Negotiation successful
0923 19:34:13.966646 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 19:34:13.968734 18281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:13.971294 18590 tablet_replica.cc:330] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: stopping tablet replica
I20260923 19:34:13.971457 18591 tablet_replica.cc:330] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: stopping tablet replica
I20260923 19:34:13.971791 18591 raft_consensus.cc:2229] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:13.971786 18590 raft_consensus.cc:2229] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:13.972335 18590 raft_consensus.cc:2258] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:13.972328 18591 raft_consensus.cc:2258] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:13.974916 18590 ts_tablet_manager.cc:1905] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:13.974907 18591 ts_tablet_manager.cc:1905] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:13.982486 18590 ts_tablet_manager.cc:1918] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:13.982482 18591 ts_tablet_manager.cc:1918] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:13.982666 18590 log.cc:1196] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/4ad4d3b3a025447e98f12200e529b1ca
I20260923 19:34:13.982666 18591 log.cc:1196] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/4ad4d3b3a025447e98f12200e529b1ca
I20260923 19:34:13.983202 18591 ts_tablet_manager.cc:1939] T 4ad4d3b3a025447e98f12200e529b1ca P e72d0b4f14944e7ab46bb3d3e7df6bfa: Deleting consensus metadata
I20260923 19:34:13.983438 18590 ts_tablet_manager.cc:1939] T 4ad4d3b3a025447e98f12200e529b1ca P adabd47e19444b5a9b4d609892cba6be: Deleting consensus metadata
I20260923 19:34:13.985337 18087 catalog_manager.cc:4841] TS adabd47e19444b5a9b4d609892cba6be (127.0.97.2:43267): tablet 4ad4d3b3a025447e98f12200e529b1ca (table test-table [id=3c0d3dd5079f4da4a7494760149fe3c7]) successfully deleted
I20260923 19:34:13.985314 18086 catalog_manager.cc:4841] TS e72d0b4f14944e7ab46bb3d3e7df6bfa (127.0.97.3:39641): tablet 4ad4d3b3a025447e98f12200e529b1ca (table test-table [id=3c0d3dd5079f4da4a7494760149fe3c7]) successfully deleted
I20260923 19:34:14.095918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 18159
I20260923 19:34:14.112442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 18290
W20260923 19:34:14.112511 18084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44403 (error 108)
I20260923 19:34:14.114539 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:14.113946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44403 (local address 127.0.0.1:59840)
0923 19:34:14.114153 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:14.114362 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 19:34:14.114833 18084 catalog_manager.cc:4584] TS 082ae47383e84f268507c797fa1c8475 (127.0.97.1:44403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ad4d3b3a025447e98f12200e529b1ca: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44403: connect: Connection refused (error 111)
I20260923 19:34:14.119660 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:14.113290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.0.97.2:43267)
0923 19:34:14.113494 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:14.113501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:14.113538 (+ 37us) server_negotiation.cc:407] Connection header received
0923 19:34:14.113605 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:14.113609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:14.113688 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:14.113783 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:14.114253 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:14.115297 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:14.116581 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:14.117005 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:14.117107 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:14.118178 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:14.118187 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:14.118204 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:14.118957 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:14.119022 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:14.119044 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:14.119154 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:14.119253 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 19:34:14.119356 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 19:34:14.120508 18412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:14.168723 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:14.168133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44403 (local address 127.0.0.1:59844)
0923 19:34:14.168436 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:14.168557 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260923 19:34:14.169842 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:14.169509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44403 (local address 127.0.0.1:59860)
0923 19:34:14.169653 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:14.169721 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 19:34:14.219385 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 18290
I20260923 19:34:14.230785 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:14.230083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44403 (local address 127.0.0.1:59874)
0923 19:34:14.230431 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:14.230558 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260923 19:34:14.231878 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:14.231562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44403 (local address 127.0.0.1:59884)
0923 19:34:14.231688 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:14.231763 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 19:34:14.234869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 18420
I20260923 19:34:14.242723 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:14.235711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.0.97.3:39641)
0923 19:34:14.235902 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:14.235908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:14.235936 (+ 28us) server_negotiation.cc:407] Connection header received
0923 19:34:14.236082 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:14.236086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:14.236216 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:14.236320 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:14.236874 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:14.237675 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:14.239081 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:14.239647 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:14.239773 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:14.240938 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:14.240948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:14.240966 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:14.241724 (+ 758us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:14.241794 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:14.241816 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:14.241916 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:14.242106 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 19:34:14.242263 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 19:34:14.243470 18542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:14.313133 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:14.312406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44403 (local address 127.0.0.1:59888)
0923 19:34:14.312719 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:14.312875 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260923 19:34:14.314601 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:14.314109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44403 (local address 127.0.0.1:59900)
0923 19:34:14.314374 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:14.314468 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260923 19:34:14.338474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 18420
I20260923 19:34:14.353507 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 18069
I20260923 19:34:14.354805 18130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:14.475977 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 18069
2026-09-23T19:34:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3088 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:14 dist-test-slave-kxfb krb5kdc[18601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:14 dist-test-slave-kxfb krb5kdc[18601](info): set up 2 sockets
Sep 23 19:34:14 dist-test-slave-kxfb krb5kdc[18601](info): commencing operation
krb5kdc: starting...
W20260923 19:34:15.518539 388 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:15 dist-test-slave-kxfb krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:34:15.798429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46255
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:16.147168 18614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:16.147486 18614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:16.147581 18614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:16.154947 18614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:16.155077 18614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:16.155126 18614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:16.155189 18614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:16.155237 18614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:16.165359 18614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:16.166714 18614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:16.167979 18614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:16.181358 18619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:16.181403 18620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:16.181358 18622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:16.182384 18614 server_base.cc:1034] running on GCE node
I20260923 19:34:16.183349 18614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:16.185178 18614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:16.186406 18614 hybrid_clock.cc:648] HybridClock initialized: now 1790192056186319 us; error 70 us; skew 500 ppm
I20260923 19:34:16.186851 18614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:16 dist-test-slave-kxfb krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:16.190403 18614 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:34:16.192098 18614 webserver.cc:460] Webserver started at http://127.0.97.62:46871/ using document root <none> and password file <none>
I20260923 19:34:16.192904 18614 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:16.193027 18614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:16.193374 18614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:16.196935 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "312bdf24ff9548f8870339e7604eceac"
format_stamp: "Formatted at 2026-09-23 19:34:16 on dist-test-slave-kxfb"
server_key: "4b999a6afef3686f4ce864d264c16366"
server_key_iv: "978c3969844f0d69b0d9696681d285a9"
server_key_version: "encryptionkey@0"
I20260923 19:34:16.197773 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "312bdf24ff9548f8870339e7604eceac"
format_stamp: "Formatted at 2026-09-23 19:34:16 on dist-test-slave-kxfb"
server_key: "4b999a6afef3686f4ce864d264c16366"
server_key_iv: "978c3969844f0d69b0d9696681d285a9"
server_key_version: "encryptionkey@0"
I20260923 19:34:16.203823 18614 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 19:34:16.208262 18629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:16.210675 18614 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:34:16.210973 18614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "312bdf24ff9548f8870339e7604eceac"
format_stamp: "Formatted at 2026-09-23 19:34:16 on dist-test-slave-kxfb"
server_key: "4b999a6afef3686f4ce864d264c16366"
server_key_iv: "978c3969844f0d69b0d9696681d285a9"
server_key_version: "encryptionkey@0"
I20260923 19:34:16.211290 18614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:16.253051 18614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:16.257819 18614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:16.258097 18614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:16.278162 18614 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46255
I20260923 19:34:16.278192 18680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46255 every 8 connection(s)
I20260923 19:34:16.279989 18614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:16.284497 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:16.290114 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 18614
I20260923 19:34:16.290380 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:16.290875 388 external_mini_cluster.cc:1428] Setting key 61b3b040d4d9424566c24ef84eeb494c
I20260923 19:34:16.297176 18681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac: Bootstrap starting.
I20260923 19:34:16.300997 18681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:16.302198 18681 log.cc:824] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:16.305920 18681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac: No bootstrap required, opened a new log
Sep 23 19:34:16 dist-test-slave-kxfb krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:16.315068 18681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312bdf24ff9548f8870339e7604eceac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46255 } }
I20260923 19:34:16.315606 18681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:16.315729 18681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312bdf24ff9548f8870339e7604eceac, State: Initialized, Role: FOLLOWER
I20260923 19:34:16.316443 18681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312bdf24ff9548f8870339e7604eceac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46255 } }
I20260923 19:34:16.316718 18681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:16.316867 18681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:16.317047 18681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:16.319952 18681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312bdf24ff9548f8870339e7604eceac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46255 } }
I20260923 19:34:16.320253 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:16.293356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.0.97.62:46255)
0923 19:34:16.294479 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:16.294503 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:16.294568 (+ 65us) server_negotiation.cc:407] Connection header received
0923 19:34:16.296997 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:16.297028 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:16.297591 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:16.298066 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:16.299664 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:16.301604 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:16.303532 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:16.304566 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:16.304789 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:16.309881 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:16.309894 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:16.309918 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:16.309989 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:16.314214 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:16.315403 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:16.315412 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:16.315420 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:16.315691 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:16.316253 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:16.316260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:16.316263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:16.317102 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:16.317481 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:16.318196 (+ 715us) server_negotiation.cc:299] Negotiation successful
0923 19:34:16.318850 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":485,"threads_started":1}
I20260923 19:34:16.320573 18681 leader_election.cc:302] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 312bdf24ff9548f8870339e7604eceac; no voters:
I20260923 19:34:16.321343 18681 leader_election.cc:288] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:16.321470 18686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:16.324031 18681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:16.324033 18686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [term 1 LEADER]: Becoming Leader. State: Replica: 312bdf24ff9548f8870339e7604eceac, State: Running, Role: LEADER
I20260923 19:34:16.324838 18686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312bdf24ff9548f8870339e7604eceac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46255 } }
I20260923 19:34:16.328032 18688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 312bdf24ff9548f8870339e7604eceac. Latest consensus state: current_term: 1 leader_uuid: "312bdf24ff9548f8870339e7604eceac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312bdf24ff9548f8870339e7604eceac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46255 } } }
I20260923 19:34:16.328572 18688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:16.329211 18687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "312bdf24ff9548f8870339e7604eceac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312bdf24ff9548f8870339e7604eceac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46255 } } }
I20260923 19:34:16.329488 18687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:16.333703 18696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:16.339517 18696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:16.349833 18696 catalog_manager.cc:1347] Generated new cluster ID: 156e6d48ed0b453ebe2eefeb555d553a
I20260923 19:34:16.349992 18696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:16.373436 18696 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:16.374904 18696 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:16.383850 18696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac: Generated new TSK 0
I20260923 19:34:16.384709 18696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:34:16.449643 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46255
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:16.779327 18707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:16.779606 18707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:16.779721 18707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:16.786978 18707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:16.787091 18707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:16.787238 18707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:34:16.796900 18707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:16.798182 18707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:16.799512 18707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:16.812525 18715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:16.812548 18712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:16.812570 18713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:16.813871 18707 server_base.cc:1034] running on GCE node
I20260923 19:34:16.814697 18707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:16.815837 18707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:16.817066 18707 hybrid_clock.cc:648] HybridClock initialized: now 1790192056816997 us; error 55 us; skew 500 ppm
I20260923 19:34:16.817451 18707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:16 dist-test-slave-kxfb krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:16.821378 18707 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:34:16.823199 18707 webserver.cc:460] Webserver started at http://127.0.97.1:46007/ using document root <none> and password file <none>
I20260923 19:34:16.824072 18707 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:16.824199 18707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:16.824563 18707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:16.828106 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "d89c9fbf8394438d89b7fb92bc0a1313"
format_stamp: "Formatted at 2026-09-23 19:34:16 on dist-test-slave-kxfb"
server_key: "15c4a10e7e0933d2ff2e33d281660c3c"
server_key_iv: "4c77baa42d5832fe81d96a106c815aba"
server_key_version: "encryptionkey@0"
I20260923 19:34:16.828977 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d89c9fbf8394438d89b7fb92bc0a1313"
format_stamp: "Formatted at 2026-09-23 19:34:16 on dist-test-slave-kxfb"
server_key: "15c4a10e7e0933d2ff2e33d281660c3c"
server_key_iv: "4c77baa42d5832fe81d96a106c815aba"
server_key_version: "encryptionkey@0"
I20260923 19:34:16.834880 18707 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 19:34:16.838788 18722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:16.840418 18707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:34:16.840619 18707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "d89c9fbf8394438d89b7fb92bc0a1313"
format_stamp: "Formatted at 2026-09-23 19:34:16 on dist-test-slave-kxfb"
server_key: "15c4a10e7e0933d2ff2e33d281660c3c"
server_key_iv: "4c77baa42d5832fe81d96a106c815aba"
server_key_version: "encryptionkey@0"
I20260923 19:34:16.840839 18707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:16.889796 18707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:16.894809 18707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:16.895133 18707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:16.896247 18707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:16.897809 18707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:16.897890 18707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:16.898025 18707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:16.898079 18707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:16.932895 18707 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33259
I20260923 19:34:16.932961 18834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33259 every 8 connection(s)
I20260923 19:34:16.934432 18707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:34:16.943554 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 18707
I20260923 19:34:16.943790 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:34:16.944303 388 external_mini_cluster.cc:1428] Setting key 3fee8b24542319f8d50419f8ab4c2616
Sep 23 19:34:16 dist-test-slave-kxfb krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:16.956297 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:16.937251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44423 (local address 127.0.97.62:46255)
0923 19:34:16.937811 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:16.937817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:16.938959 (+ 1142us) server_negotiation.cc:407] Connection header received
0923 19:34:16.940894 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:16.940899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:16.940984 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:16.941108 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:16.942687 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:16.943743 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:16.945639 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:16.946272 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:16.946432 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:16.950191 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:16.950196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:16.950200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:16.950234 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:16.953005 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:16.953798 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:16.953804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:16.953808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:16.953939 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:16.954385 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:16.954391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:16.954394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:16.954689 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:16.954828 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:16.955607 (+ 779us) server_negotiation.cc:299] Negotiation successful
0923 19:34:16.955896 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":259,"threads_started":1}
I20260923 19:34:16.957427 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:16.937929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46255 (local address 127.0.97.1:44423)
0923 19:34:16.938773 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:16.938824 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:16.940610 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:16.941315 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:16.941323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:16.941866 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:16.942509 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:16.942527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:16.943949 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:16.943953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:16.945499 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:16.945512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:16.946559 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:16.946579 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:16.946842 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:16.947550 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:34:16.947585 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:16.949949 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:16.953181 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:16.953192 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:16.953209 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:16.953679 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:16.954042 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:16.954047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:16.954050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:16.954247 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:16.954816 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:16.954825 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:16.955205 (+ 380us) client_negotiation.cc:769] Sending connection context
0923 19:34:16.955460 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 19:34:16.956088 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":339,"threads_started":1}
I20260923 19:34:16.959307 18835 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46255
I20260923 19:34:16.959731 18835 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:16.960768 18835 heartbeater.cc:511] Master 127.0.97.62:46255 requested a full tablet report, sending...
I20260923 19:34:16.963400 18646 ts_manager.cc:194] Registered new tserver with Master: d89c9fbf8394438d89b7fb92bc0a1313 (127.0.97.1:33259)
I20260923 19:34:16.965534 18646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44423
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:34:17.001551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46255
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:17.329552 18841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:17.329869 18841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:17.329988 18841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:17.337281 18841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:17.337399 18841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:17.337546 18841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:34:17.348131 18841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:17.349418 18841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:17.350703 18841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:17.364233 18849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:17.364513 18847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:17.364804 18846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:17.367805 18841 server_base.cc:1034] running on GCE node
I20260923 19:34:17.368484 18841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:17.369500 18841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:17.370726 18841 hybrid_clock.cc:648] HybridClock initialized: now 1790192057370653 us; error 59 us; skew 500 ppm
I20260923 19:34:17.371088 18841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:17 dist-test-slave-kxfb krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:17.374524 18841 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:34:17.376195 18841 webserver.cc:460] Webserver started at http://127.0.97.2:33367/ using document root <none> and password file <none>
I20260923 19:34:17.376909 18841 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:17.377020 18841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:17.377313 18841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:17.380191 18841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "144e3c59a98f40ea8c9006e64f456836"
format_stamp: "Formatted at 2026-09-23 19:34:17 on dist-test-slave-kxfb"
server_key: "6226c56905bb138bf20a61863ab62f34"
server_key_iv: "699e94ade8897c2173460aa244152f40"
server_key_version: "encryptionkey@0"
I20260923 19:34:17.380940 18841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "144e3c59a98f40ea8c9006e64f456836"
format_stamp: "Formatted at 2026-09-23 19:34:17 on dist-test-slave-kxfb"
server_key: "6226c56905bb138bf20a61863ab62f34"
server_key_iv: "699e94ade8897c2173460aa244152f40"
server_key_version: "encryptionkey@0"
I20260923 19:34:17.386092 18841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:34:17.389874 18856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:17.391508 18841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:34:17.391741 18841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "144e3c59a98f40ea8c9006e64f456836"
format_stamp: "Formatted at 2026-09-23 19:34:17 on dist-test-slave-kxfb"
server_key: "6226c56905bb138bf20a61863ab62f34"
server_key_iv: "699e94ade8897c2173460aa244152f40"
server_key_version: "encryptionkey@0"
I20260923 19:34:17.391999 18841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:17.410720 18841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:17.414813 18841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:17.415067 18841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:17.416028 18841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:17.417382 18841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:17.417454 18841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:17.417547 18841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:17.417598 18841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:17.451664 18841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39459
I20260923 19:34:17.451748 18968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39459 every 8 connection(s)
I20260923 19:34:17.453002 18841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:34:17.455096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 18841
I20260923 19:34:17.455298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:34:17.455874 388 external_mini_cluster.cc:1428] Setting key 480cef432f9139a1d8204bac109c051e
Sep 23 19:34:17 dist-test-slave-kxfb krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:17.477516 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:17.456129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50355 (local address 127.0.97.62:46255)
0923 19:34:17.456336 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:17.456340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:17.457773 (+ 1433us) server_negotiation.cc:407] Connection header received
0923 19:34:17.460580 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:17.460587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:17.460674 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:17.460823 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:17.463114 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:17.463906 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:17.466176 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:17.466649 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:17.466749 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:17.471135 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:17.471142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:17.471147 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:17.471189 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:17.474395 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:17.475107 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:17.475112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:17.475116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:17.475269 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:17.475692 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:17.475696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:17.475698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:17.476013 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:17.476158 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:17.476984 (+ 826us) server_negotiation.cc:299] Negotiation successful
0923 19:34:17.477198 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 19:34:17.478731 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:17.456701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46255 (local address 127.0.97.2:50355)
0923 19:34:17.457603 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:17.457648 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:17.460193 (+ 2545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:17.461146 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:17.461161 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:17.461969 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:17.462915 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:17.462940 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:17.464081 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:17.464087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:17.466006 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:17.466025 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:17.466893 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:17.466900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:17.467185 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:17.467983 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:34:17.468019 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:17.470862 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:17.474527 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:17.474535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:17.474550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:17.474979 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:17.475382 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:17.475386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:17.475388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:17.475579 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:17.476190 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:17.476197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:17.476569 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 19:34:17.476845 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 19:34:17.477318 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":401,"threads_started":1}
I20260923 19:34:17.480458 18969 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46255
I20260923 19:34:17.480794 18969 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:17.481618 18969 heartbeater.cc:511] Master 127.0.97.62:46255 requested a full tablet report, sending...
I20260923 19:34:17.483350 18646 ts_manager.cc:194] Registered new tserver with Master: 144e3c59a98f40ea8c9006e64f456836 (127.0.97.2:39459)
I20260923 19:34:17.484252 18646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50355
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:34:17.519328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46255
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:17.837615 18974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:17.837935 18974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:17.838066 18974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:17.845428 18974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:17.845554 18974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:17.845702 18974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:34:17.855485 18974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:17.856791 18974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:17.858078 18974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:17.871815 18979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:17.871770 18982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:17.871870 18980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:17.875844 18974 server_base.cc:1034] running on GCE node
I20260923 19:34:17.876520 18974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:17.877521 18974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:17.878726 18974 hybrid_clock.cc:648] HybridClock initialized: now 1790192057878662 us; error 54 us; skew 500 ppm
I20260923 19:34:17.879070 18974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:17 dist-test-slave-kxfb krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:17.882365 18974 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:34:17.884006 18974 webserver.cc:460] Webserver started at http://127.0.97.3:46291/ using document root <none> and password file <none>
I20260923 19:34:17.884671 18974 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:17.884785 18974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:17.885056 18974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:17.887915 18974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "ed056f180a5940879122e8b0197c6df8"
format_stamp: "Formatted at 2026-09-23 19:34:17 on dist-test-slave-kxfb"
server_key: "0f213c6673fbf64c662d2dea019a3b43"
server_key_iv: "c98d9e30da67f12fcd5507839ad017dd"
server_key_version: "encryptionkey@0"
I20260923 19:34:17.888680 18974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed056f180a5940879122e8b0197c6df8"
format_stamp: "Formatted at 2026-09-23 19:34:17 on dist-test-slave-kxfb"
server_key: "0f213c6673fbf64c662d2dea019a3b43"
server_key_iv: "c98d9e30da67f12fcd5507839ad017dd"
server_key_version: "encryptionkey@0"
I20260923 19:34:17.894124 18974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 19:34:17.898079 18989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:17.899617 18974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:34:17.899770 18974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "ed056f180a5940879122e8b0197c6df8"
format_stamp: "Formatted at 2026-09-23 19:34:17 on dist-test-slave-kxfb"
server_key: "0f213c6673fbf64c662d2dea019a3b43"
server_key_iv: "c98d9e30da67f12fcd5507839ad017dd"
server_key_version: "encryptionkey@0"
I20260923 19:34:17.900007 18974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:17.920414 18974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:17.924680 18974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:17.924952 18974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:17.926005 18974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:17.927372 18974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:17.927453 18974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:17.927577 18974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:17.927631 18974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:17.962002 18974 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44985
I20260923 19:34:17.962080 19101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44985 every 8 connection(s)
I20260923 19:34:17.963276 18974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:34:17.966849 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 18974
I20260923 19:34:17.967073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:34:17.967645 388 external_mini_cluster.cc:1428] Setting key 250b164c59d1dc664c0707c02bb01169
I20260923 19:34:17.969076 18835 heartbeater.cc:503] Master 127.0.97.62:46255 was elected leader, sending a full tablet report...
Sep 23 19:34:17 dist-test-slave-kxfb krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:17.985210 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:17.966256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51769 (local address 127.0.97.62:46255)
0923 19:34:17.966481 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:17.966486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:17.967952 (+ 1466us) server_negotiation.cc:407] Connection header received
0923 19:34:17.970636 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:17.970641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:17.970729 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:17.970839 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:17.972550 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:17.973303 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:17.975018 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:17.975407 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:17.975516 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:17.979054 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:17.979059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:17.979062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:17.979094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:17.981361 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:17.982534 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:17.982543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:17.982547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:17.982774 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:17.983306 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:17.983311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:17.983314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:17.983747 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:17.983906 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:17.984669 (+ 763us) server_negotiation.cc:299] Negotiation successful
0923 19:34:17.984903 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 19:34:17.985958 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:17.966838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46255 (local address 127.0.97.3:51769)
0923 19:34:17.967751 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:17.967812 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:17.970270 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:17.971140 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:17.971150 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:17.971732 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:17.972379 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:17.972397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:17.973450 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:17.973455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:17.974860 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:17.974873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:17.975607 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:17.975612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:17.975809 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:17.976494 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:34:17.976523 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:17.978845 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:17.981536 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:17.981545 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:17.981559 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:17.982186 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:17.982944 (+ 758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:17.982949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:17.982953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:17.983170 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:17.983887 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:17.983893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:17.984251 (+ 358us) client_negotiation.cc:769] Sending connection context
0923 19:34:17.984484 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 19:34:17.984904 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":355,"threads_started":1}
I20260923 19:34:17.987519 19102 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46255
I20260923 19:34:17.987845 19102 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:17.988730 19102 heartbeater.cc:511] Master 127.0.97.62:46255 requested a full tablet report, sending...
I20260923 19:34:17.990469 18645 ts_manager.cc:194] Registered new tserver with Master: ed056f180a5940879122e8b0197c6df8 (127.0.97.3:44985)
I20260923 19:34:17.991465 18645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51769
I20260923 19:34:17.996259 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:34:18.015616 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.000998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.0.97.62:46255)
0923 19:34:18.001261 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.001268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.001629 (+ 361us) server_negotiation.cc:407] Connection header received
0923 19:34:18.001807 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.001813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.001917 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.002038 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:18.003669 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.004600 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.005951 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.006441 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.006544 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.009224 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:18.009231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:18.009236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:18.009283 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:18.012283 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:18.013189 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:18.013193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:18.013196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:18.013334 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:18.013820 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:18.013824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:18.013826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:18.014413 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:18.014674 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.015061 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.015239 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 19:34:18.020649 18645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:34:18.028983 18645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:34:18.056519 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.043448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.0.97.2:39459)
0923 19:34:18.046418 (+ 2970us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.046428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.046463 (+ 35us) server_negotiation.cc:407] Connection header received
0923 19:34:18.046585 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.046593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.046920 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.048071 (+ 1151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:18.048341 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.050149 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.053443 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.054963 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.055129 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.055366 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.055953 (+ 587us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.056187 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2775,"thread_start_us":2589,"threads_started":1}
I20260923 19:34:18.057077 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.043798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.97.1:33259)
0923 19:34:18.045848 (+ 2050us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.045859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.046816 (+ 957us) server_negotiation.cc:407] Connection header received
0923 19:34:18.047162 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.047172 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.047571 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.047828 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:18.048881 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.050580 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.053753 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.055333 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.055467 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.055722 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.056148 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.056386 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1850,"thread_start_us":1670,"threads_started":1}
I20260923 19:34:18.057143 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.043164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39459 (local address 127.0.0.1:38674)
0923 19:34:18.044764 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:18.044821 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:18.045135 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:18.047207 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:18.047216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:18.047265 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:18.047835 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.047850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.050335 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.050342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.053263 (+ 2921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.053284 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.055202 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.055209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.055473 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.055674 (+ 201us) client_negotiation.cc:769] Sending connection context
0923 19:34:18.055832 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 19:34:18.056104 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"mutex_wait_us":164,"thread_start_us":3262,"threads_started":1}
I20260923 19:34:18.058136 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.043313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33259 (local address 127.0.0.1:37060)
0923 19:34:18.046683 (+ 3370us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:18.046703 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:18.046953 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:18.048166 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:18.048172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:18.048204 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:18.048662 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.048677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.050778 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.050797 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.053588 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.053598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.055547 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.055554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.055863 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.055905 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 19:34:18.057566 (+ 1661us) client_negotiation.cc:240] Negotiation successful
0923 19:34:18.057742 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3239,"mutex_wait_us":53,"thread_start_us":937,"threads_started":1}
I20260923 19:34:18.061216 18904 tablet_service.cc:1467] Processing CreateTablet for tablet 14427b57d46f4e50938663475622d9c2 (DEFAULT_TABLE table=test-table [id=c271905889be49f4b9220dd6a7d5d503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:18.062716 18904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14427b57d46f4e50938663475622d9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:18.062687 18770 tablet_service.cc:1467] Processing CreateTablet for tablet 14427b57d46f4e50938663475622d9c2 (DEFAULT_TABLE table=test-table [id=c271905889be49f4b9220dd6a7d5d503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:18.064334 18770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14427b57d46f4e50938663475622d9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:18.064951 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.047189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.0.97.3:44985)
0923 19:34:18.054111 (+ 6922us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.054120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.054450 (+ 330us) server_negotiation.cc:407] Connection header received
0923 19:34:18.054584 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.054593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.054927 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.055156 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:18.056416 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.059210 (+ 2794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.061640 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.063802 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.063966 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.064198 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.064453 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.064625 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6715,"thread_start_us":6544,"threads_started":1}
I20260923 19:34:18.065267 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.046885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44985 (local address 127.0.0.1:49456)
0923 19:34:18.054241 (+ 7356us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:18.054266 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:18.054430 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:18.055454 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:18.055460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:18.055498 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:18.056172 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.056185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.059366 (+ 3181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.059372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.061379 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.061395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.064045 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.064052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.064240 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.064280 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 19:34:18.064476 (+ 196us) client_negotiation.cc:240] Negotiation successful
0923 19:34:18.064636 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7222,"mutex_wait_us":64,"thread_start_us":6451,"threads_started":1}
I20260923 19:34:18.069456 19037 tablet_service.cc:1467] Processing CreateTablet for tablet 14427b57d46f4e50938663475622d9c2 (DEFAULT_TABLE table=test-table [id=c271905889be49f4b9220dd6a7d5d503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:18.070937 19037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14427b57d46f4e50938663475622d9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:18.078023 19121 tablet_bootstrap.cc:492] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: Bootstrap starting.
I20260923 19:34:18.078830 19120 tablet_bootstrap.cc:492] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: Bootstrap starting.
I20260923 19:34:18.081832 19121 tablet_bootstrap.cc:654] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:18.082154 19120 tablet_bootstrap.cc:654] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:18.083151 19121 log.cc:824] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:18.083241 19120 log.cc:824] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:18.085150 19124 tablet_bootstrap.cc:492] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: Bootstrap starting.
I20260923 19:34:18.086849 19120 tablet_bootstrap.cc:492] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: No bootstrap required, opened a new log
I20260923 19:34:18.086953 19121 tablet_bootstrap.cc:492] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: No bootstrap required, opened a new log
I20260923 19:34:18.087124 19120 ts_tablet_manager.cc:1397] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 19:34:18.087257 19121 ts_tablet_manager.cc:1397] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260923 19:34:18.089041 19124 tablet_bootstrap.cc:654] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:18.090461 19124 log.cc:824] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:18.093890 19124 tablet_bootstrap.cc:492] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: No bootstrap required, opened a new log
I20260923 19:34:18.094213 19124 ts_tablet_manager.cc:1397] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 19:34:18.094502 19120 raft_consensus.cc:348] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } }
I20260923 19:34:18.094861 19120 raft_consensus.cc:374] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:18.094946 19120 raft_consensus.cc:729] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144e3c59a98f40ea8c9006e64f456836, State: Initialized, Role: FOLLOWER
I20260923 19:34:18.095470 19120 consensus_queue.cc:260] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } }
I20260923 19:34:18.096550 18969 heartbeater.cc:503] Master 127.0.97.62:46255 was elected leader, sending a full tablet report...
I20260923 19:34:18.097170 19120 ts_tablet_manager.cc:1428] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260923 19:34:18.098011 19121 raft_consensus.cc:348] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } }
I20260923 19:34:18.098528 19121 raft_consensus.cc:374] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:18.098636 19121 raft_consensus.cc:729] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89c9fbf8394438d89b7fb92bc0a1313, State: Initialized, Role: FOLLOWER
I20260923 19:34:18.099403 19121 consensus_queue.cc:260] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } }
I20260923 19:34:18.101454 19121 ts_tablet_manager.cc:1428] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260923 19:34:18.104800 19124 raft_consensus.cc:348] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } }
I20260923 19:34:18.105240 19124 raft_consensus.cc:374] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:18.105353 19124 raft_consensus.cc:729] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed056f180a5940879122e8b0197c6df8, State: Initialized, Role: FOLLOWER
I20260923 19:34:18.105926 19124 consensus_queue.cc:260] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } }
I20260923 19:34:18.107007 19102 heartbeater.cc:503] Master 127.0.97.62:46255 was elected leader, sending a full tablet report...
I20260923 19:34:18.107467 19124 ts_tablet_manager.cc:1428] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260923 19:34:18.164928 19128 raft_consensus.cc:482] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:34:18.165257 19128 raft_consensus.cc:504] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } }
I20260923 19:34:18.167017 19128 leader_election.cc:288] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 144e3c59a98f40ea8c9006e64f456836 (127.0.97.2:39459), d89c9fbf8394438d89b7fb92bc0a1313 (127.0.97.1:33259)
I20260923 19:34:18.173095 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.167396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39459 (local address 127.0.97.3:55547)
0923 19:34:18.167797 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:18.167819 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:18.167992 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:18.168337 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:18.168342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:18.168368 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:18.168594 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.168605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.169634 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.169640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.171117 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.171129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.172282 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.172288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.172491 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.172529 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 19:34:18.172633 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 19:34:18.172787 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":3}
I20260923 19:34:18.173112 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.167496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55547 (local address 127.0.97.2:39459)
0923 19:34:18.167729 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.167735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.167926 (+ 191us) server_negotiation.cc:407] Connection header received
0923 19:34:18.168141 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.168147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.168222 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.168299 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:18.168729 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.169501 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.171298 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.172095 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.172240 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.172347 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.172681 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.172823 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 19:34:18.173981 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.167580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46131 (local address 127.0.97.1:33259)
0923 19:34:18.167812 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.167819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.168427 (+ 608us) server_negotiation.cc:407] Connection header received
0923 19:34:18.168671 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.168676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.168743 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.168816 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:18.169345 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.170050 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.171830 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.172897 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.173016 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.173129 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.173603 (+ 474us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.173728 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 19:34:18.174588 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.167549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33259 (local address 127.0.97.3:46131)
0923 19:34:18.168299 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:18.168315 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:18.168522 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:18.168852 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:18.168857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:18.168889 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:18.169200 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.169208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.170191 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.170195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.171657 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:18.171669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.173267 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.173272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:18.173427 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.173456 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 19:34:18.174167 (+ 711us) client_negotiation.cc:240] Negotiation successful
0923 19:34:18.174277 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"spinlock_wait_cycles":22272,"thread_start_us":443,"threads_started":1}
I20260923 19:34:18.174472 18924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14427b57d46f4e50938663475622d9c2" candidate_uuid: "ed056f180a5940879122e8b0197c6df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144e3c59a98f40ea8c9006e64f456836" is_pre_election: true
I20260923 19:34:18.175073 18924 raft_consensus.cc:2454] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed056f180a5940879122e8b0197c6df8 in term 0.
I20260923 19:34:18.175515 18790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14427b57d46f4e50938663475622d9c2" candidate_uuid: "ed056f180a5940879122e8b0197c6df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" is_pre_election: true
I20260923 19:34:18.175762 18993 leader_election.cc:302] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 144e3c59a98f40ea8c9006e64f456836, ed056f180a5940879122e8b0197c6df8; no voters:
I20260923 19:34:18.176023 18790 raft_consensus.cc:2454] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed056f180a5940879122e8b0197c6df8 in term 0.
I20260923 19:34:18.176200 19128 raft_consensus.cc:2781] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:34:18.176368 19128 raft_consensus.cc:482] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:34:18.176478 19128 raft_consensus.cc:3037] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:18.178803 19128 raft_consensus.cc:504] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } }
I20260923 19:34:18.179400 19128 leader_election.cc:288] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [CANDIDATE]: Term 1 election: Requested vote from peers 144e3c59a98f40ea8c9006e64f456836 (127.0.97.2:39459), d89c9fbf8394438d89b7fb92bc0a1313 (127.0.97.1:33259)
I20260923 19:34:18.179883 18790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14427b57d46f4e50938663475622d9c2" candidate_uuid: "ed056f180a5940879122e8b0197c6df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d89c9fbf8394438d89b7fb92bc0a1313"
I20260923 19:34:18.179888 18924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14427b57d46f4e50938663475622d9c2" candidate_uuid: "ed056f180a5940879122e8b0197c6df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144e3c59a98f40ea8c9006e64f456836"
I20260923 19:34:18.180061 18790 raft_consensus.cc:3037] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:18.180087 18924 raft_consensus.cc:3037] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:18.182322 18790 raft_consensus.cc:2454] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed056f180a5940879122e8b0197c6df8 in term 1.
I20260923 19:34:18.182638 18924 raft_consensus.cc:2454] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed056f180a5940879122e8b0197c6df8 in term 1.
I20260923 19:34:18.182701 18991 leader_election.cc:302] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d89c9fbf8394438d89b7fb92bc0a1313, ed056f180a5940879122e8b0197c6df8; no voters:
I20260923 19:34:18.183102 19128 raft_consensus.cc:2781] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:18.183594 19128 raft_consensus.cc:686] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 1 LEADER]: Becoming Leader. State: Replica: ed056f180a5940879122e8b0197c6df8, State: Running, Role: LEADER
I20260923 19:34:18.184175 19128 consensus_queue.cc:237] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } }
W20260923 19:34:18.187175 18836 tablet.cc:2369] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:34:18.189688 18646 catalog_manager.cc:5495] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed056f180a5940879122e8b0197c6df8 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ed056f180a5940879122e8b0197c6df8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed056f180a5940879122e8b0197c6df8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44985 } health_report { overall_health: HEALTHY } } }
W20260923 19:34:18.205034 18970 tablet.cc:2369] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:34:18.214895 19103 tablet.cc:2369] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:34:18.217262 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.211771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.0.97.3:44985)
0923 19:34:18.211946 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.211950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.211970 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:34:18.212116 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.212119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.212197 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.212348 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:34:18.212693 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.213431 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.214911 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.215516 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.215649 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.216120 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:34:18.216584 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:34:18.216712 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.217001 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.217042 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 19:34:18.226812 18790 raft_consensus.cc:1261] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [term 1 FOLLOWER]: Refusing update from remote peer ed056f180a5940879122e8b0197c6df8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:18.227996 19130 consensus_queue.cc:1035] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d89c9fbf8394438d89b7fb92bc0a1313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:18.228345 18924 raft_consensus.cc:1261] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [term 1 FOLLOWER]: Refusing update from remote peer ed056f180a5940879122e8b0197c6df8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:18.229187 19130 consensus_queue.cc:1035] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "144e3c59a98f40ea8c9006e64f456836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:18.258458 18646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:18.258721 18646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:18.262524 18646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 312bdf24ff9548f8870339e7604eceac: Sending DeleteTablet for 3 replicas of tablet 14427b57d46f4e50938663475622d9c2
I20260923 19:34:18.263795 18770 tablet_service.cc:1514] Processing DeleteTablet for tablet 14427b57d46f4e50938663475622d9c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37060
I20260923 19:34:18.263972 18904 tablet_service.cc:1514] Processing DeleteTablet for tablet 14427b57d46f4e50938663475622d9c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38674
I20260923 19:34:18.265302 19037 tablet_service.cc:1514] Processing DeleteTablet for tablet 14427b57d46f4e50938663475622d9c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49456
I20260923 19:34:18.267058 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 18707
I20260923 19:34:18.273526 19143 tablet_replica.cc:330] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: stopping tablet replica
I20260923 19:34:18.274027 19143 raft_consensus.cc:2229] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:18.274623 19143 raft_consensus.cc:2258] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:18.275518 19145 tablet_replica.cc:330] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: stopping tablet replica
I20260923 19:34:18.275942 19145 raft_consensus.cc:2229] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:34:18.276664 19145 raft_consensus.cc:2258] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:18.277446 19144 tablet_replica.cc:330] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: stopping tablet replica
I20260923 19:34:18.277841 19144 raft_consensus.cc:2229] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:18.277927 19143 ts_tablet_manager.cc:1905] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:18.280200 19145 ts_tablet_manager.cc:1905] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:18.280284 19144 raft_consensus.cc:2258] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 19:34:18 dist-test-slave-kxfb krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:34:18.285816 19144 ts_tablet_manager.cc:1905] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:18.286969 19143 ts_tablet_manager.cc:1918] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:18.287127 19145 ts_tablet_manager.cc:1918] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:18.287294 19145 log.cc:1196] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/14427b57d46f4e50938663475622d9c2
I20260923 19:34:18.287320 19143 log.cc:1196] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/14427b57d46f4e50938663475622d9c2
I20260923 19:34:18.287761 19145 ts_tablet_manager.cc:1939] T 14427b57d46f4e50938663475622d9c2 P ed056f180a5940879122e8b0197c6df8: Deleting consensus metadata
I20260923 19:34:18.288072 19143 ts_tablet_manager.cc:1939] T 14427b57d46f4e50938663475622d9c2 P d89c9fbf8394438d89b7fb92bc0a1313: Deleting consensus metadata
I20260923 19:34:18.289598 18633 catalog_manager.cc:4841] TS ed056f180a5940879122e8b0197c6df8 (127.0.97.3:44985): tablet 14427b57d46f4e50938663475622d9c2 (table test-table [id=c271905889be49f4b9220dd6a7d5d503]) successfully deleted
I20260923 19:34:18.290238 18631 catalog_manager.cc:4841] TS d89c9fbf8394438d89b7fb92bc0a1313 (127.0.97.1:33259): tablet 14427b57d46f4e50938663475622d9c2 (table test-table [id=c271905889be49f4b9220dd6a7d5d503]) successfully deleted
I20260923 19:34:18.292374 19144 ts_tablet_manager.cc:1918] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:18.292521 19144 log.cc:1196] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/14427b57d46f4e50938663475622d9c2
I20260923 19:34:18.293000 19144 ts_tablet_manager.cc:1939] T 14427b57d46f4e50938663475622d9c2 P 144e3c59a98f40ea8c9006e64f456836: Deleting consensus metadata
I20260923 19:34:18.294761 18633 catalog_manager.cc:4841] TS 144e3c59a98f40ea8c9006e64f456836 (127.0.97.2:39459): tablet 14427b57d46f4e50938663475622d9c2 (table test-table [id=c271905889be49f4b9220dd6a7d5d503]) successfully deleted
I20260923 19:34:18.298192 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.268254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.0.97.1:33259)
0923 19:34:18.268498 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.268505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.273180 (+ 4675us) server_negotiation.cc:407] Connection header received
0923 19:34:18.273575 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.273580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.273681 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.273760 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:18.275710 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.276736 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.278792 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.279389 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.279488 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.291390 (+ 11902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:18.291411 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:18.291429 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:18.291485 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:18.295351 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:18.296003 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:18.296010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:18.296012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:18.296136 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:18.296504 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:18.296507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:18.296509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:18.296906 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:18.297261 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.297462 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.297665 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 19:34:18.298870 18830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:18.387322 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 18707
I20260923 19:34:18.403420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 18841
Sep 23 19:34:18 dist-test-slave-kxfb krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:34:18.422662 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.404309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.0.97.2:39459)
0923 19:34:18.404713 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.404721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.404747 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:34:18.404836 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.404842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.404958 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.405086 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:18.406419 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.407626 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.408913 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.409577 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.409720 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.415553 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:18.415572 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:18.415589 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:18.415643 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:18.419530 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:18.420334 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:18.420345 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:18.420350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:18.420532 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:18.420886 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:18.420891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:18.420894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:18.421450 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:18.421786 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.421950 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.422220 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260923 19:34:18.423586 18964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:18.512148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 18841
I20260923 19:34:18.530735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 18974
Sep 23 19:34:18 dist-test-slave-kxfb krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:34:18.547366 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:18.531594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.0.97.3:44985)
0923 19:34:18.531822 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:18.531829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:18.531847 (+ 18us) server_negotiation.cc:407] Connection header received
0923 19:34:18.531914 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:18.531918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:18.531992 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:18.532106 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:18.533385 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.534321 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.535552 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:18.536028 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:18.536135 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:18.540595 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:18.540610 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:18.540631 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:18.540670 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:18.544064 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:18.545208 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:18.545214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:18.545217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:18.545380 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:18.545792 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:18.545795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:18.545797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:18.546170 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:18.546470 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:18.546856 (+ 386us) server_negotiation.cc:299] Negotiation successful
0923 19:34:18.547065 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 19:34:18.548409 19097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:18.670998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 18974
I20260923 19:34:18.688275 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 18614
I20260923 19:34:18.689527 18676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:18.801221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 18614
2026-09-23T19:34:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4326 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T19:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:18Z Disabled control of system clock
I20260923 19:34:18.881961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34455
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 19:34:19.204160 19154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:19.204490 19154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:19.204572 19154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:19.211886 19154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:19.212018 19154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:19.212054 19154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:19.212101 19154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:19.212138 19154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:19.221939 19154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:19.223255 19154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:19.225025 19154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:19.239650 19159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:19.239773 19160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:19.239799 19162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:19.240918 19154 server_base.cc:1034] running on GCE node
I20260923 19:34:19.241909 19154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:19.243662 19154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:19.244905 19154 hybrid_clock.cc:648] HybridClock initialized: now 1790192059244846 us; error 47 us; skew 500 ppm
I20260923 19:34:19.245343 19154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:19.247983 19154 webserver.cc:460] Webserver started at http://127.0.97.62:40559/ using document root <none> and password file <none>
I20260923 19:34:19.248773 19154 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:19.248879 19154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:19.249230 19154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:19.252527 19154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "3b2061a604d94f4e83e30efde4e2e5e0"
format_stamp: "Formatted at 2026-09-23 19:34:19 on dist-test-slave-kxfb"
server_key: "e4424ec8d8818961e99368ec76741226"
server_key_iv: "f5d5c2e783a10d0f1509674117cffe48"
server_key_version: "encryptionkey@0"
I20260923 19:34:19.253374 19154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "3b2061a604d94f4e83e30efde4e2e5e0"
format_stamp: "Formatted at 2026-09-23 19:34:19 on dist-test-slave-kxfb"
server_key: "e4424ec8d8818961e99368ec76741226"
server_key_iv: "f5d5c2e783a10d0f1509674117cffe48"
server_key_version: "encryptionkey@0"
I20260923 19:34:19.259310 19154 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 19:34:19.263705 19168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:19.265444 19154 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:34:19.265666 19154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "3b2061a604d94f4e83e30efde4e2e5e0"
format_stamp: "Formatted at 2026-09-23 19:34:19 on dist-test-slave-kxfb"
server_key: "e4424ec8d8818961e99368ec76741226"
server_key_iv: "f5d5c2e783a10d0f1509674117cffe48"
server_key_version: "encryptionkey@0"
I20260923 19:34:19.265916 19154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:19.296864 19154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:19.297971 19154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:19.298324 19154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:19.319459 19154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34455
I20260923 19:34:19.319507 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34455 every 8 connection(s)
I20260923 19:34:19.321098 19154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:19.325801 19220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:19.328037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 19154
I20260923 19:34:19.328204 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:19.328675 388 external_mini_cluster.cc:1428] Setting key ce6864e2f2aba34bc3b942c65c5e380c
I20260923 19:34:19.339432 19220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0: Bootstrap starting.
I20260923 19:34:19.343817 19220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:19.344777 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:19.331030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.0.97.62:34455)
0923 19:34:19.332221 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:19.332234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:19.332282 (+ 48us) server_negotiation.cc:407] Connection header received
0923 19:34:19.334222 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:19.334246 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:19.334821 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:19.335190 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:19.335854 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:19.337741 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:19.339732 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:19.340508 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:19.340652 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:19.342094 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:19.342107 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:19.342129 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:19.342813 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:19.342933 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:19.342957 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:19.343056 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:19.343280 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 19:34:19.343634 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":432,"threads_started":1}
I20260923 19:34:19.345145 19220 log.cc:824] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:19.348591 19220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0: No bootstrap required, opened a new log
I20260923 19:34:19.356202 19220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2061a604d94f4e83e30efde4e2e5e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34455 } }
I20260923 19:34:19.356575 19220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:19.356675 19220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2061a604d94f4e83e30efde4e2e5e0, State: Initialized, Role: FOLLOWER
I20260923 19:34:19.357267 19220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2061a604d94f4e83e30efde4e2e5e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34455 } }
I20260923 19:34:19.357530 19220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:19.357697 19220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:19.357888 19220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:19.360217 19220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2061a604d94f4e83e30efde4e2e5e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34455 } }
I20260923 19:34:19.360745 19220 leader_election.cc:302] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b2061a604d94f4e83e30efde4e2e5e0; no voters:
I20260923 19:34:19.361371 19220 leader_election.cc:288] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:19.361621 19225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:19.362514 19225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 3b2061a604d94f4e83e30efde4e2e5e0, State: Running, Role: LEADER
I20260923 19:34:19.363310 19225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2061a604d94f4e83e30efde4e2e5e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34455 } }
I20260923 19:34:19.363575 19220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:19.367098 19226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b2061a604d94f4e83e30efde4e2e5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2061a604d94f4e83e30efde4e2e5e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34455 } } }
I20260923 19:34:19.367074 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2061a604d94f4e83e30efde4e2e5e0. Latest consensus state: current_term: 1 leader_uuid: "3b2061a604d94f4e83e30efde4e2e5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2061a604d94f4e83e30efde4e2e5e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34455 } } }
I20260923 19:34:19.367704 19226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:19.367807 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:19.370116 19234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:19.375378 19234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:19.384732 19234 catalog_manager.cc:1347] Generated new cluster ID: b2570aa21f464660957408a31fdde446
I20260923 19:34:19.384886 19234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:19.405907 19234 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:19.407917 19234 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:19.431965 19234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0: Generated new TSK 0
I20260923 19:34:19.433082 19234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:19.447505 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34455
--builtin_ntp_servers=127.0.97.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 19:34:19.824241 19244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:19.824671 19244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:19.824812 19244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:19.832780 19244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:19.832917 19244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:19.833074 19244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:34:19.842868 19244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:19.844698 19244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:19.846930 19244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:19.861541 19249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:19.861603 19250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:19.861603 19252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:19.862578 19244 server_base.cc:1034] running on GCE node
I20260923 19:34:19.863349 19244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:19.864558 19244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:19.865782 19244 hybrid_clock.cc:648] HybridClock initialized: now 1790192059865725 us; error 49 us; skew 500 ppm
I20260923 19:34:19.866173 19244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:19.869107 19244 webserver.cc:460] Webserver started at http://127.0.97.1:35851/ using document root <none> and password file <none>
I20260923 19:34:19.869923 19244 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:19.870029 19244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:19.870455 19244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:19.873749 19244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "cbfbdab175db4710a3cf6d17f6e5b9df"
format_stamp: "Formatted at 2026-09-23 19:34:19 on dist-test-slave-kxfb"
server_key: "5f38cc27c5f08fafc79706621d51e16a"
server_key_iv: "35cd1f3eb2037686ac32105c8270e8d0"
server_key_version: "encryptionkey@0"
I20260923 19:34:19.874632 19244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cbfbdab175db4710a3cf6d17f6e5b9df"
format_stamp: "Formatted at 2026-09-23 19:34:19 on dist-test-slave-kxfb"
server_key: "5f38cc27c5f08fafc79706621d51e16a"
server_key_iv: "35cd1f3eb2037686ac32105c8270e8d0"
server_key_version: "encryptionkey@0"
I20260923 19:34:19.880481 19244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:34:19.884408 19258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:19.886101 19244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 19:34:19.886341 19244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "cbfbdab175db4710a3cf6d17f6e5b9df"
format_stamp: "Formatted at 2026-09-23 19:34:19 on dist-test-slave-kxfb"
server_key: "5f38cc27c5f08fafc79706621d51e16a"
server_key_iv: "35cd1f3eb2037686ac32105c8270e8d0"
server_key_version: "encryptionkey@0"
I20260923 19:34:19.886579 19244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:19.906169 19244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:19.907203 19244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:19.907485 19244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:19.908705 19244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:19.910540 19244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:19.910806 19244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:19.911023 19244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:19.911113 19244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:19.946098 19244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39895
I20260923 19:34:19.946396 19370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39895 every 8 connection(s)
I20260923 19:34:19.947573 19244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:34:19.957213 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 19244
I20260923 19:34:19.957387 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:34:19.957826 388 external_mini_cluster.cc:1428] Setting key 7512e60defdaa585edbd2c48377bcb40
I20260923 19:34:19.961436 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34455
--builtin_ntp_servers=127.0.97.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 19:34:19.962518 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:19.950602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57187 (local address 127.0.97.62:34455)
0923 19:34:19.951131 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:19.951137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:19.952511 (+ 1374us) server_negotiation.cc:407] Connection header received
0923 19:34:19.954619 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:19.954626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:19.954729 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:19.954835 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:19.956132 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:19.956950 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:19.958943 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:19.959521 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:19.959651 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:19.961451 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:19.961458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:19.961463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:19.961488 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:19.961549 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:19.961567 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:19.961680 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:19.961952 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 19:34:19.962075 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
I20260923 19:34:19.963475 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:19.951203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34455 (local address 127.0.97.1:57187)
0923 19:34:19.952313 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:19.952362 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:19.954268 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:19.955122 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:19.955132 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:19.955249 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:19.955948 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:19.955966 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:19.957119 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:19.957123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:19.958800 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:19.958815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:19.959893 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:19.959906 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:19.960249 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:19.961051 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:34:19.961054 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:19.961100 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:34:19.961204 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:19.961728 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:19.961738 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:19.961747 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 19:34:19.961917 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 19:34:19.962279 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":289,"threads_started":1}
I20260923 19:34:19.965204 19371 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34455
I20260923 19:34:19.965628 19371 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:19.966768 19371 heartbeater.cc:511] Master 127.0.97.62:34455 requested a full tablet report, sending...
I20260923 19:34:19.969525 19185 ts_manager.cc:194] Registered new tserver with Master: cbfbdab175db4710a3cf6d17f6e5b9df (127.0.97.1:39895)
I20260923 19:34:19.971642 19185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:57187
W20260923 19:34:20.310928 19375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:20.311204 19375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:20.311309 19375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:20.318519 19375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:20.318645 19375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:20.318821 19375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:34:20.328230 19375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:20.329560 19375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:20.331352 19375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:20.345777 19381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:20.346071 19383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:20.346161 19380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:20.346954 19375 server_base.cc:1034] running on GCE node
I20260923 19:34:20.347617 19375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:20.348639 19375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:20.349884 19375 hybrid_clock.cc:648] HybridClock initialized: now 1790192060349827 us; error 50 us; skew 500 ppm
I20260923 19:34:20.350242 19375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:20.352735 19375 webserver.cc:460] Webserver started at http://127.0.97.2:39875/ using document root <none> and password file <none>
I20260923 19:34:20.353451 19375 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:20.353550 19375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:20.353859 19375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:20.356830 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "6af932b34bf4499483d59b6fd40da05d"
format_stamp: "Formatted at 2026-09-23 19:34:20 on dist-test-slave-kxfb"
server_key: "96f243680d091240ae49cc530b916250"
server_key_iv: "6d6cb805e6c4cac198d1b8812a33304a"
server_key_version: "encryptionkey@0"
I20260923 19:34:20.357635 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6af932b34bf4499483d59b6fd40da05d"
format_stamp: "Formatted at 2026-09-23 19:34:20 on dist-test-slave-kxfb"
server_key: "96f243680d091240ae49cc530b916250"
server_key_iv: "6d6cb805e6c4cac198d1b8812a33304a"
server_key_version: "encryptionkey@0"
I20260923 19:34:20.363447 19375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 19:34:20.367411 19389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:20.369040 19375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:34:20.369207 19375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "6af932b34bf4499483d59b6fd40da05d"
format_stamp: "Formatted at 2026-09-23 19:34:20 on dist-test-slave-kxfb"
server_key: "96f243680d091240ae49cc530b916250"
server_key_iv: "6d6cb805e6c4cac198d1b8812a33304a"
server_key_version: "encryptionkey@0"
I20260923 19:34:20.369412 19375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:20.397853 19375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:20.398849 19375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:20.399128 19375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:20.400352 19375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:20.401893 19375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:20.401981 19375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:20.402087 19375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:20.402158 19375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:20.435516 19375 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39521
I20260923 19:34:20.435652 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39521 every 8 connection(s)
I20260923 19:34:20.436766 19375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:34:20.443751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 19375
I20260923 19:34:20.443989 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:34:20.444602 388 external_mini_cluster.cc:1428] Setting key bcd869422723386a8463e67921bb487a
I20260923 19:34:20.448566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34455
--builtin_ntp_servers=127.0.97.20:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 19:34:20.453063 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.439747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36927 (local address 127.0.97.62:34455)
0923 19:34:20.439977 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:20.439983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:20.441522 (+ 1539us) server_negotiation.cc:407] Connection header received
0923 19:34:20.443976 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:20.443983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:20.444093 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:20.444211 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:20.445851 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.446932 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.449434 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.450094 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.450250 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.452121 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:20.452128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:20.452134 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:20.452162 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:20.452210 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:20.452228 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:20.452341 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:20.452608 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 19:34:20.452728 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 19:34:20.454627 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.440164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34455 (local address 127.0.97.2:36927)
0923 19:34:20.441321 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:20.441371 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:20.443499 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:20.444550 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:20.444566 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:20.444712 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:20.445609 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.445639 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.447119 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.447123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.449196 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.449215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.450409 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.450416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.450733 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.451684 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:34:20.451688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:20.451754 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:34:20.451878 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:20.452348 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:20.452357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:20.452365 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 19:34:20.452590 (+ 225us) client_negotiation.cc:240] Negotiation successful
0923 19:34:20.453064 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":333,"threads_started":1}
I20260923 19:34:20.456218 19502 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34455
I20260923 19:34:20.456615 19502 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:20.457449 19502 heartbeater.cc:511] Master 127.0.97.62:34455 requested a full tablet report, sending...
I20260923 19:34:20.459005 19185 ts_manager.cc:194] Registered new tserver with Master: 6af932b34bf4499483d59b6fd40da05d (127.0.97.2:39521)
I20260923 19:34:20.459846 19185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:36927
W20260923 19:34:20.768805 19505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:20.769119 19505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:20.769227 19505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:20.776798 19505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:20.776924 19505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:20.777050 19505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:34:20.786243 19505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:20.787513 19505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:20.789247 19505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:20.803188 19510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:20.803148 19513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:20.803270 19511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:20.804453 19505 server_base.cc:1034] running on GCE node
I20260923 19:34:20.805107 19505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:20.806142 19505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:20.807349 19505 hybrid_clock.cc:648] HybridClock initialized: now 1790192060807295 us; error 54 us; skew 500 ppm
I20260923 19:34:20.807672 19505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:20.809976 19505 webserver.cc:460] Webserver started at http://127.0.97.3:46483/ using document root <none> and password file <none>
I20260923 19:34:20.810707 19505 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:20.810817 19505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:20.811118 19505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:20.813922 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "4f8a5791c3b44467bd35df5c62d23a34"
format_stamp: "Formatted at 2026-09-23 19:34:20 on dist-test-slave-kxfb"
server_key: "8700fda468077cc809625a5f4bbd4ca0"
server_key_iv: "f36252d70aa56713d7aad76ad8af2d01"
server_key_version: "encryptionkey@0"
I20260923 19:34:20.814689 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4f8a5791c3b44467bd35df5c62d23a34"
format_stamp: "Formatted at 2026-09-23 19:34:20 on dist-test-slave-kxfb"
server_key: "8700fda468077cc809625a5f4bbd4ca0"
server_key_iv: "f36252d70aa56713d7aad76ad8af2d01"
server_key_version: "encryptionkey@0"
I20260923 19:34:20.819747 19505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260923 19:34:20.823455 19519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:20.824994 19505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:34:20.825148 19505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "4f8a5791c3b44467bd35df5c62d23a34"
format_stamp: "Formatted at 2026-09-23 19:34:20 on dist-test-slave-kxfb"
server_key: "8700fda468077cc809625a5f4bbd4ca0"
server_key_iv: "f36252d70aa56713d7aad76ad8af2d01"
server_key_version: "encryptionkey@0"
I20260923 19:34:20.825347 19505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:20.854542 19505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:20.855381 19505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:20.855630 19505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:20.856695 19505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:20.858261 19505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:20.858377 19505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 19:34:20.858486 19505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:20.858551 19505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:20.891930 19505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35471
I20260923 19:34:20.892068 19631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35471 every 8 connection(s)
I20260923 19:34:20.893199 19505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:34:20.899292 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 19505
I20260923 19:34:20.899477 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:34:20.899932 388 external_mini_cluster.cc:1428] Setting key ad2ad78e422d56e2234870756197668a
I20260923 19:34:20.905977 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.896197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46099 (local address 127.0.97.62:34455)
0923 19:34:20.896415 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:20.896421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:20.897969 (+ 1548us) server_negotiation.cc:407] Connection header received
0923 19:34:20.899878 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:20.899884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:20.899970 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:20.900062 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:20.901086 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.901848 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.903503 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.903899 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.904002 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.905202 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:20.905208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:20.905212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:20.905229 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:20.905275 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:20.905286 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:20.905356 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:20.905559 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 19:34:20.905653 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 19:34:20.906950 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.896755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34455 (local address 127.0.97.3:46099)
0923 19:34:20.897788 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:20.897834 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:20.899610 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:20.900286 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:20.900294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:20.900379 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:20.900928 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.900944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.901989 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.901993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.903368 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.903379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.904110 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.904117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.904310 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.904909 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:34:20.904912 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:20.904949 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:34:20.905020 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:20.905371 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:20.905382 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:20.905389 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 19:34:20.905534 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 19:34:20.905897 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":318,"threads_started":1}
I20260923 19:34:20.908313 19632 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34455
I20260923 19:34:20.908726 19632 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:20.909660 19632 heartbeater.cc:511] Master 127.0.97.62:34455 requested a full tablet report, sending...
I20260923 19:34:20.911327 19185 ts_manager.cc:194] Registered new tserver with Master: 4f8a5791c3b44467bd35df5c62d23a34 (127.0.97.3:35471)
I20260923 19:34:20.912137 19185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46099
I20260923 19:34:20.915233 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:34:20.926178 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.919706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.0.97.62:34455)
0923 19:34:20.919908 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:20.919915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:20.920359 (+ 444us) server_negotiation.cc:407] Connection header received
0923 19:34:20.920550 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:20.920555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:20.920633 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:20.920724 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:20.921300 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.922077 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.923572 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.924044 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.924161 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.925425 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:20.925430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:20.925433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:20.925456 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:20.925493 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:20.925506 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:20.925644 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:20.925804 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 19:34:20.925869 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 19:34:20.930698 19185 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:34:20.938177 19185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:34:20.975175 19371 heartbeater.cc:503] Master 127.0.97.62:34455 was elected leader, sending a full tablet report...
I20260923 19:34:20.989531 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.953180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39521 (local address 127.0.0.1:52448)
0923 19:34:20.971776 (+ 18596us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:20.971841 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:20.979053 (+ 7212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:20.980005 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:20.980010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:20.980044 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:20.980960 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.980976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.983351 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.983362 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.985953 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.985972 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.988406 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.988417 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.988661 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.988748 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 19:34:20.988882 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 19:34:20.989106 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18159,"mutex_wait_us":92,"thread_start_us":21787,"threads_started":1}
I20260923 19:34:20.989604 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.953740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.0.97.1:39895)
0923 19:34:20.966734 (+ 12994us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:20.966752 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:20.975830 (+ 9078us) server_negotiation.cc:407] Connection header received
0923 19:34:20.975963 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:20.975973 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:20.976553 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:20.977615 (+ 1062us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:20.978371 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.980034 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.986248 (+ 6214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.987860 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.987979 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.988270 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:20.989061 (+ 791us) server_negotiation.cc:299] Negotiation successful
0923 19:34:20.989229 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12809,"thread_start_us":12683,"threads_started":1}
I20260923 19:34:20.990445 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.953967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.0.97.2:39521)
0923 19:34:20.974694 (+ 20727us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:20.974705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:20.974756 (+ 51us) server_negotiation.cc:407] Connection header received
0923 19:34:20.979289 (+ 4533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:20.979297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:20.979524 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:20.979725 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:20.981253 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.982358 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.986314 (+ 3956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.987860 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.987996 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.988243 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:20.989872 (+ 1629us) server_negotiation.cc:299] Negotiation successful
0923 19:34:20.990106 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20548,"thread_start_us":20418,"threads_started":1}
I20260923 19:34:20.990846 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.953459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39895 (local address 127.0.0.1:41610)
0923 19:34:20.975256 (+ 21797us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:20.975278 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:20.975470 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:20.976902 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:20.976910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:20.976945 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:20.978083 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.978100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.980265 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.980272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.985974 (+ 5702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.985983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.988431 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.988436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.988668 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.988748 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 19:34:20.988882 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 19:34:20.990469 (+ 1587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21659,"mutex_wait_us":1565,"spinlock_wait_cycles":3968,"thread_start_us":19647,"threads_started":1}
I20260923 19:34:20.994318 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.954603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.0.97.3:35471)
0923 19:34:20.975300 (+ 20697us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:20.975311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:20.976878 (+ 1567us) server_negotiation.cc:407] Connection header received
0923 19:34:20.977305 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:20.977314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:20.977668 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:20.977867 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:20.978813 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.980977 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.987375 (+ 6398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.989006 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.989274 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.989654 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:20.993549 (+ 3895us) server_negotiation.cc:299] Negotiation successful
0923 19:34:20.993799 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20499,"thread_start_us":20689,"threads_started":1}
I20260923 19:34:20.995173 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:20.953884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35471 (local address 127.0.0.1:36248)
0923 19:34:20.976722 (+ 22838us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:20.976769 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:20.976992 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:20.978203 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:20.978208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:20.978233 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:20.978647 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.978656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.981157 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.981163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.987121 (+ 5958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:20.987137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:20.989824 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:20.989832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:20.990090 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:20.990133 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 19:34:20.990238 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 19:34:20.994717 (+ 4479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22644,"mutex_wait_us":3,"thread_start_us":22872,"threads_started":1}
I20260923 19:34:20.995486 19306 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae69023b05a4531b3d7a43706d1b2f5 (DEFAULT_TABLE table=test-table [id=1a7cda9a39714f19ba1adfcc835fb8f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:20.995282 19437 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae69023b05a4531b3d7a43706d1b2f5 (DEFAULT_TABLE table=test-table [id=1a7cda9a39714f19ba1adfcc835fb8f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:20.996824 19437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae69023b05a4531b3d7a43706d1b2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:20.999689 19567 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae69023b05a4531b3d7a43706d1b2f5 (DEFAULT_TABLE table=test-table [id=1a7cda9a39714f19ba1adfcc835fb8f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:21.001220 19567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae69023b05a4531b3d7a43706d1b2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:21.002755 19306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae69023b05a4531b3d7a43706d1b2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:21.018567 19650 tablet_bootstrap.cc:492] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: Bootstrap starting.
I20260923 19:34:21.022168 19651 tablet_bootstrap.cc:492] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: Bootstrap starting.
I20260923 19:34:21.024801 19650 tablet_bootstrap.cc:654] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:21.026058 19650 log.cc:824] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:21.027046 19651 tablet_bootstrap.cc:654] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:21.028401 19651 log.cc:824] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:21.038962 19651 tablet_bootstrap.cc:492] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: No bootstrap required, opened a new log
I20260923 19:34:21.039386 19651 ts_tablet_manager.cc:1397] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260923 19:34:21.041615 19652 tablet_bootstrap.cc:492] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: Bootstrap starting.
I20260923 19:34:21.046811 19650 tablet_bootstrap.cc:492] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: No bootstrap required, opened a new log
I20260923 19:34:21.047175 19650 ts_tablet_manager.cc:1397] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.015s
I20260923 19:34:21.047278 19652 tablet_bootstrap.cc:654] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:21.048691 19652 log.cc:824] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:21.049239 19651 raft_consensus.cc:348] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } }
I20260923 19:34:21.049870 19651 raft_consensus.cc:374] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:21.050006 19651 raft_consensus.cc:729] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8a5791c3b44467bd35df5c62d23a34, State: Initialized, Role: FOLLOWER
I20260923 19:34:21.050881 19651 consensus_queue.cc:260] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } }
I20260923 19:34:21.056577 19651 ts_tablet_manager.cc:1428] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260923 19:34:21.057010 19632 heartbeater.cc:503] Master 127.0.97.62:34455 was elected leader, sending a full tablet report...
I20260923 19:34:21.060114 19650 raft_consensus.cc:348] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } }
I20260923 19:34:21.060747 19650 raft_consensus.cc:374] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:21.060851 19650 raft_consensus.cc:729] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af932b34bf4499483d59b6fd40da05d, State: Initialized, Role: FOLLOWER
I20260923 19:34:21.061719 19650 consensus_queue.cc:260] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } }
I20260923 19:34:21.066229 19502 heartbeater.cc:503] Master 127.0.97.62:34455 was elected leader, sending a full tablet report...
I20260923 19:34:21.067090 19650 ts_tablet_manager.cc:1428] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: Time spent starting tablet: real 0.020s user 0.008s sys 0.011s
I20260923 19:34:21.068218 19652 tablet_bootstrap.cc:492] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: No bootstrap required, opened a new log
I20260923 19:34:21.068573 19652 ts_tablet_manager.cc:1397] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.011s
I20260923 19:34:21.077661 19652 raft_consensus.cc:348] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } }
I20260923 19:34:21.078176 19652 raft_consensus.cc:374] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:21.078305 19652 raft_consensus.cc:729] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfbdab175db4710a3cf6d17f6e5b9df, State: Initialized, Role: FOLLOWER
I20260923 19:34:21.079092 19652 consensus_queue.cc:260] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } }
I20260923 19:34:21.082692 19652 ts_tablet_manager.cc:1428] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 19:34:21.084095 19657 raft_consensus.cc:482] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:34:21.084458 19657 raft_consensus.cc:504] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } }
I20260923 19:34:21.086629 19657 leader_election.cc:288] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbfbdab175db4710a3cf6d17f6e5b9df (127.0.97.1:39895), 4f8a5791c3b44467bd35df5c62d23a34 (127.0.97.3:35471)
I20260923 19:34:21.095505 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:21.087593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58133 (local address 127.0.97.3:35471)
0923 19:34:21.087839 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:21.087847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:21.087876 (+ 29us) server_negotiation.cc:407] Connection header received
0923 19:34:21.088175 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:21.088183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:21.088285 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:21.088351 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:21.089254 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.090397 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.093515 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.094549 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.094683 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:21.094803 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:21.095083 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 19:34:21.095223 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 19:34:21.095580 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:21.086969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35471 (local address 127.0.97.2:58133)
0923 19:34:21.087746 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:21.087769 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:21.087930 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:21.088562 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:21.088567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:21.088593 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:21.089020 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:21.089032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.090559 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.090568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:21.093267 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:21.093285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.094760 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.094766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:21.094923 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:21.094951 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 19:34:21.095036 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 19:34:21.095208 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":920,"threads_started":1}
I20260923 19:34:21.096323 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:21.087560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50105 (local address 127.0.97.1:39895)
0923 19:34:21.087783 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:21.087789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:21.089755 (+ 1966us) server_negotiation.cc:407] Connection header received
0923 19:34:21.089911 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:21.089917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:21.090012 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:21.090138 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:21.091101 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.091804 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.093712 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.095072 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.095222 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:21.095341 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:21.095903 (+ 562us) server_negotiation.cc:299] Negotiation successful
0923 19:34:21.096075 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 19:34:21.096446 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:21.087000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39895 (local address 127.0.97.2:50105)
0923 19:34:21.089640 (+ 2640us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:21.089659 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:21.089787 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:21.090338 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:21.090344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:21.090379 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:21.090945 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:21.090958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.091950 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.091955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:21.093478 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:21.093493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.095496 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.095503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:21.095713 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:21.095753 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 19:34:21.095863 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 19:34:21.096037 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2510,"mutex_wait_us":28,"thread_start_us":2317,"threads_started":1}
I20260923 19:34:21.097234 19587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae69023b05a4531b3d7a43706d1b2f5" candidate_uuid: "6af932b34bf4499483d59b6fd40da05d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8a5791c3b44467bd35df5c62d23a34" is_pre_election: true
I20260923 19:34:21.097498 19326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae69023b05a4531b3d7a43706d1b2f5" candidate_uuid: "6af932b34bf4499483d59b6fd40da05d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" is_pre_election: true
I20260923 19:34:21.097864 19587 raft_consensus.cc:2454] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af932b34bf4499483d59b6fd40da05d in term 0.
I20260923 19:34:21.098120 19326 raft_consensus.cc:2454] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af932b34bf4499483d59b6fd40da05d in term 0.
I20260923 19:34:21.098794 19393 leader_election.cc:302] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f8a5791c3b44467bd35df5c62d23a34, 6af932b34bf4499483d59b6fd40da05d, cbfbdab175db4710a3cf6d17f6e5b9df; no voters:
I20260923 19:34:21.099308 19657 raft_consensus.cc:2781] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:34:21.099493 19657 raft_consensus.cc:482] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:34:21.099606 19657 raft_consensus.cc:3037] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:21.103081 19657 raft_consensus.cc:504] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } }
I20260923 19:34:21.104156 19657 leader_election.cc:288] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [CANDIDATE]: Term 1 election: Requested vote from peers cbfbdab175db4710a3cf6d17f6e5b9df (127.0.97.1:39895), 4f8a5791c3b44467bd35df5c62d23a34 (127.0.97.3:35471)
I20260923 19:34:21.104807 19326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae69023b05a4531b3d7a43706d1b2f5" candidate_uuid: "6af932b34bf4499483d59b6fd40da05d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df"
I20260923 19:34:21.104810 19587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae69023b05a4531b3d7a43706d1b2f5" candidate_uuid: "6af932b34bf4499483d59b6fd40da05d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8a5791c3b44467bd35df5c62d23a34"
I20260923 19:34:21.105093 19326 raft_consensus.cc:3037] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:21.105078 19587 raft_consensus.cc:3037] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:21.108636 19326 raft_consensus.cc:2454] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af932b34bf4499483d59b6fd40da05d in term 1.
I20260923 19:34:21.108636 19587 raft_consensus.cc:2454] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af932b34bf4499483d59b6fd40da05d in term 1.
I20260923 19:34:21.109159 19391 leader_election.cc:302] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af932b34bf4499483d59b6fd40da05d, cbfbdab175db4710a3cf6d17f6e5b9df; no voters:
I20260923 19:34:21.109640 19657 raft_consensus.cc:2781] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:21.110213 19657 raft_consensus.cc:686] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 1 LEADER]: Becoming Leader. State: Replica: 6af932b34bf4499483d59b6fd40da05d, State: Running, Role: LEADER
I20260923 19:34:21.110826 19657 consensus_queue.cc:237] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } }
I20260923 19:34:21.116633 19184 catalog_manager.cc:5495] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af932b34bf4499483d59b6fd40da05d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6af932b34bf4499483d59b6fd40da05d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af932b34bf4499483d59b6fd40da05d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260923 19:34:21.145117 19633 tablet.cc:2369] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:34:21.155727 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:21.149094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.0.97.2:39521)
0923 19:34:21.149385 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:21.149392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:21.149419 (+ 27us) server_negotiation.cc:407] Connection header received
0923 19:34:21.149648 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:21.149654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:21.149780 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:21.149953 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:34:21.150410 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.151499 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.152933 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.153598 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.153731 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:21.154237 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:34:21.154977 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:34:21.155177 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:21.155364 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 19:34:21.155416 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 19:34:21.167032 19587 raft_consensus.cc:1261] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [term 1 FOLLOWER]: Refusing update from remote peer 6af932b34bf4499483d59b6fd40da05d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:21.167322 19326 raft_consensus.cc:1261] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [term 1 FOLLOWER]: Refusing update from remote peer 6af932b34bf4499483d59b6fd40da05d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:21.168249 19657 consensus_queue.cc:1035] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8a5791c3b44467bd35df5c62d23a34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:21.168769 19661 consensus_queue.cc:1035] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbfbdab175db4710a3cf6d17f6e5b9df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:21.191723 19666 mvcc.cc:205] Tried to move back new op lower bound from 7332626682526789632 to 7332626682314383360. Current Snapshot: MvccSnapshot[applied={T|T < 7332626682526789632}]
I20260923 19:34:21.221437 19185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:21.221880 19185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:21.227212 19185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b2061a604d94f4e83e30efde4e2e5e0: Sending DeleteTablet for 3 replicas of tablet 2ae69023b05a4531b3d7a43706d1b2f5
I20260923 19:34:21.228899 19437 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae69023b05a4531b3d7a43706d1b2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:21 UTC) from {username='slave'} at 127.0.0.1:52448
I20260923 19:34:21.228993 19306 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae69023b05a4531b3d7a43706d1b2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:21 UTC) from {username='slave'} at 127.0.0.1:41610
I20260923 19:34:21.230744 19567 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae69023b05a4531b3d7a43706d1b2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:21 UTC) from {username='slave'} at 127.0.0.1:36248
I20260923 19:34:21.240350 19673 tablet_replica.cc:330] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: stopping tablet replica
I20260923 19:34:21.241063 19673 raft_consensus.cc:2229] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:34:21.242012 19673 raft_consensus.cc:2258] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:21.244010 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 19244
I20260923 19:34:21.256699 19673 ts_tablet_manager.cc:1905] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:21.260131 19675 tablet_replica.cc:330] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: stopping tablet replica
I20260923 19:34:21.260846 19675 raft_consensus.cc:2229] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:21.261526 19674 tablet_replica.cc:330] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: stopping tablet replica
I20260923 19:34:21.262137 19675 raft_consensus.cc:2258] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:21.262692 19674 raft_consensus.cc:2229] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:21.265012 19674 raft_consensus.cc:2258] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:21.266256 19673 ts_tablet_manager.cc:1918] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:21.266499 19673 log.cc:1196] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/2ae69023b05a4531b3d7a43706d1b2f5
I20260923 19:34:21.267861 19674 ts_tablet_manager.cc:1905] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:21.268275 19675 ts_tablet_manager.cc:1905] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:21.269304 19673 ts_tablet_manager.cc:1939] T 2ae69023b05a4531b3d7a43706d1b2f5 P 6af932b34bf4499483d59b6fd40da05d: Deleting consensus metadata
I20260923 19:34:21.273335 19169 catalog_manager.cc:4841] TS 6af932b34bf4499483d59b6fd40da05d (127.0.97.2:39521): tablet 2ae69023b05a4531b3d7a43706d1b2f5 (table test-table [id=1a7cda9a39714f19ba1adfcc835fb8f7]) successfully deleted
I20260923 19:34:21.278590 19675 ts_tablet_manager.cc:1918] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:21.280404 19675 log.cc:1196] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/2ae69023b05a4531b3d7a43706d1b2f5
I20260923 19:34:21.283350 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:21.245838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.0.97.1:39895)
0923 19:34:21.246052 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:21.246060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:21.271155 (+ 25095us) server_negotiation.cc:407] Connection header received
0923 19:34:21.273827 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:21.273835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:21.273970 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:21.274068 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:21.274970 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.276214 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.278476 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.279180 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.279290 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:21.281054 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:21.281067 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:21.281091 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:21.282132 (+ 1041us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:21.282210 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:21.282243 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:21.282407 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:21.282668 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 19:34:21.282823 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 19:34:21.284797 19366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:21.285260 19675 ts_tablet_manager.cc:1939] T 2ae69023b05a4531b3d7a43706d1b2f5 P 4f8a5791c3b44467bd35df5c62d23a34: Deleting consensus metadata
I20260923 19:34:21.287753 19172 catalog_manager.cc:4841] TS 4f8a5791c3b44467bd35df5c62d23a34 (127.0.97.3:35471): tablet 2ae69023b05a4531b3d7a43706d1b2f5 (table test-table [id=1a7cda9a39714f19ba1adfcc835fb8f7]) successfully deleted
I20260923 19:34:21.438939 19674 ts_tablet_manager.cc:1918] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:21.439271 19674 log.cc:1196] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/2ae69023b05a4531b3d7a43706d1b2f5
I20260923 19:34:21.439821 19674 ts_tablet_manager.cc:1939] T 2ae69023b05a4531b3d7a43706d1b2f5 P cbfbdab175db4710a3cf6d17f6e5b9df: Deleting consensus metadata
I20260923 19:34:21.441875 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 19244
I20260923 19:34:21.442019 19170 catalog_manager.cc:4841] TS cbfbdab175db4710a3cf6d17f6e5b9df (127.0.97.1:39895): tablet 2ae69023b05a4531b3d7a43706d1b2f5 (table test-table [id=1a7cda9a39714f19ba1adfcc835fb8f7]) successfully deleted
I20260923 19:34:21.460109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 19375
I20260923 19:34:21.468328 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:21.460969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.0.97.2:39521)
0923 19:34:21.461203 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:21.461214 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:21.461277 (+ 63us) server_negotiation.cc:407] Connection header received
0923 19:34:21.461496 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:21.461505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:21.461621 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:21.461734 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:21.462163 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.463101 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.464595 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.465177 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.465292 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:21.466473 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:21.466482 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:21.466501 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:21.467272 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:21.467324 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:21.467348 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:21.467443 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:21.467625 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 19:34:21.467737 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 19:34:21.469146 19497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:21.585170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 19375
I20260923 19:34:21.604038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 19505
I20260923 19:34:21.612007 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:21.604987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.0.97.3:35471)
0923 19:34:21.605578 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:21.605583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:21.605605 (+ 22us) server_negotiation.cc:407] Connection header received
0923 19:34:21.605673 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:21.605677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:21.605768 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:21.605876 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:21.606439 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.607202 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.608901 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:21.609294 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:21.609391 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:21.610653 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:21.610661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:21.610677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:21.611248 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:21.611298 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:21.611318 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:21.611420 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:21.611567 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 19:34:21.611658 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":335,"threads_started":1}
I20260923 19:34:21.612855 19627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:21.713160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 19505
I20260923 19:34:21.728457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 19154
I20260923 19:34:21.729714 19215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:21.839653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 19154
2026-09-23T19:34:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3040 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:21 dist-test-slave-kxfb krb5kdc[19684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:21 dist-test-slave-kxfb krb5kdc[19684](info): set up 2 sockets
Sep 23 19:34:21 dist-test-slave-kxfb krb5kdc[19684](info): commencing operation
krb5kdc: starting...
W20260923 19:34:22.886855 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:23 dist-test-slave-kxfb krb5kdc[19684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:34:23.178508 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45521
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:23.513217 19697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:23.513501 19697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:23.513609 19697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:23.520901 19697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:23.521034 19697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:23.521080 19697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:23.521138 19697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:23.521203 19697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:23.531380 19697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:23.532716 19697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:23.533980 19697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:23.545910 19702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:23.546077 19705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:23.546077 19703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:23.547118 19697 server_base.cc:1034] running on GCE node
I20260923 19:34:23.547919 19697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:23.549307 19697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:23.550523 19697 hybrid_clock.cc:648] HybridClock initialized: now 1790192063550465 us; error 56 us; skew 500 ppm
I20260923 19:34:23.550901 19697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:23 dist-test-slave-kxfb krb5kdc[19684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:23.554212 19697 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:34:23.556103 19697 webserver.cc:460] Webserver started at http://127.0.97.62:36587/ using document root <none> and password file <none>
I20260923 19:34:23.556937 19697 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:23.557051 19697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:23.557355 19697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:23.560966 19697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "5c7a0164f4274bfe8d45fcc6b3208498"
format_stamp: "Formatted at 2026-09-23 19:34:23 on dist-test-slave-kxfb"
server_key: "e5c465bd0f49605c2da0d521e2f02950"
server_key_iv: "bc28add76103aff21cee8fe0c339512b"
server_key_version: "encryptionkey@0"
I20260923 19:34:23.561807 19697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "5c7a0164f4274bfe8d45fcc6b3208498"
format_stamp: "Formatted at 2026-09-23 19:34:23 on dist-test-slave-kxfb"
server_key: "e5c465bd0f49605c2da0d521e2f02950"
server_key_iv: "bc28add76103aff21cee8fe0c339512b"
server_key_version: "encryptionkey@0"
I20260923 19:34:23.568184 19697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 19:34:23.572579 19712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:23.574554 19697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:34:23.574779 19697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "5c7a0164f4274bfe8d45fcc6b3208498"
format_stamp: "Formatted at 2026-09-23 19:34:23 on dist-test-slave-kxfb"
server_key: "e5c465bd0f49605c2da0d521e2f02950"
server_key_iv: "bc28add76103aff21cee8fe0c339512b"
server_key_version: "encryptionkey@0"
I20260923 19:34:23.575021 19697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:23.590467 19697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:23.595767 19697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:23.596081 19697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:23.617223 19697 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45521
I20260923 19:34:23.617262 19763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45521 every 8 connection(s)
I20260923 19:34:23.618975 19697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:23.619776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 19697
I20260923 19:34:23.619971 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:23.620529 388 external_mini_cluster.cc:1428] Setting key cfee4f9725634a76078aff0bc8da037a
I20260923 19:34:23.624023 19764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:23.637197 19764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498: Bootstrap starting.
Sep 23 19:34:23 dist-test-slave-kxfb krb5kdc[19684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:23.640552 19764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:23.641614 19764 log.cc:824] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:23.644919 19764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498: No bootstrap required, opened a new log
I20260923 19:34:23.648386 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:23.622851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.0.97.62:45521)
0923 19:34:23.623832 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:23.623847 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:23.623899 (+ 52us) server_negotiation.cc:407] Connection header received
0923 19:34:23.625912 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:23.625942 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:23.626549 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:23.627034 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:23.628875 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:23.630635 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:23.632529 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:23.633582 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:23.633778 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:23.639091 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:23.639108 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:23.639130 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:23.639191 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:23.643198 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:23.644179 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:23.644187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:23.644195 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:23.644396 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:23.644814 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:23.644818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:23.644820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:23.645551 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:23.645914 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:23.646467 (+ 553us) server_negotiation.cc:299] Negotiation successful
0923 19:34:23.647043 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":705,"thread_start_us":468,"threads_started":1}
I20260923 19:34:23.653546 19764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7a0164f4274bfe8d45fcc6b3208498" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45521 } }
I20260923 19:34:23.653905 19764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:23.653996 19764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7a0164f4274bfe8d45fcc6b3208498, State: Initialized, Role: FOLLOWER
I20260923 19:34:23.654561 19764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7a0164f4274bfe8d45fcc6b3208498" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45521 } }
I20260923 19:34:23.654771 19764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:23.654896 19764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:23.655053 19764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:23.657480 19764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7a0164f4274bfe8d45fcc6b3208498" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45521 } }
I20260923 19:34:23.658048 19764 leader_election.cc:302] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c7a0164f4274bfe8d45fcc6b3208498; no voters:
I20260923 19:34:23.658761 19764 leader_election.cc:288] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:23.659015 19769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:23.659883 19769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [term 1 LEADER]: Becoming Leader. State: Replica: 5c7a0164f4274bfe8d45fcc6b3208498, State: Running, Role: LEADER
I20260923 19:34:23.660597 19769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7a0164f4274bfe8d45fcc6b3208498" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45521 } }
I20260923 19:34:23.660887 19764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:23.665928 19771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c7a0164f4274bfe8d45fcc6b3208498. Latest consensus state: current_term: 1 leader_uuid: "5c7a0164f4274bfe8d45fcc6b3208498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7a0164f4274bfe8d45fcc6b3208498" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45521 } } }
I20260923 19:34:23.666394 19771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:23.666832 19770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c7a0164f4274bfe8d45fcc6b3208498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7a0164f4274bfe8d45fcc6b3208498" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45521 } } }
I20260923 19:34:23.673341 19770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:23.676039 19779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:23.682032 19779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:23.693712 19779 catalog_manager.cc:1347] Generated new cluster ID: db78f6e8e18940b797518b728cc2da3c
I20260923 19:34:23.693848 19779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:23.719591 19779 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:23.721342 19779 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:23.731174 19779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498: Generated new TSK 0
I20260923 19:34:23.732366 19779 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 19:34:23.805985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45521
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:24.166595 19790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:24.166908 19790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:24.167037 19790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:24.174486 19790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:24.174604 19790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:24.174741 19790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:34:24.185305 19790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:24.186683 19790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:24.188001 19790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:24.202651 19798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:24.202760 19796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:24.202777 19795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:24.203502 19790 server_base.cc:1034] running on GCE node
I20260923 19:34:24.204408 19790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:24.205830 19790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:24.207114 19790 hybrid_clock.cc:648] HybridClock initialized: now 1790192064207029 us; error 59 us; skew 500 ppm
I20260923 19:34:24.207569 19790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:24 dist-test-slave-kxfb krb5kdc[19684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:24.211642 19790 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 19:34:24.213639 19790 webserver.cc:460] Webserver started at http://127.0.97.1:33855/ using document root <none> and password file <none>
I20260923 19:34:24.214637 19790 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:24.214793 19790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:24.215180 19790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:24.219038 19790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "85e592a47f7f43cd9f653d5adf428ddc"
format_stamp: "Formatted at 2026-09-23 19:34:24 on dist-test-slave-kxfb"
server_key: "ce08622559bb6cb65702c841df66f427"
server_key_iv: "5e7abcf09c76978d252aae815e45dc65"
server_key_version: "encryptionkey@0"
I20260923 19:34:24.220091 19790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "85e592a47f7f43cd9f653d5adf428ddc"
format_stamp: "Formatted at 2026-09-23 19:34:24 on dist-test-slave-kxfb"
server_key: "ce08622559bb6cb65702c841df66f427"
server_key_iv: "5e7abcf09c76978d252aae815e45dc65"
server_key_version: "encryptionkey@0"
I20260923 19:34:24.227650 19790 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260923 19:34:24.232610 19805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:24.234576 19790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 19:34:24.234818 19790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "85e592a47f7f43cd9f653d5adf428ddc"
format_stamp: "Formatted at 2026-09-23 19:34:24 on dist-test-slave-kxfb"
server_key: "ce08622559bb6cb65702c841df66f427"
server_key_iv: "5e7abcf09c76978d252aae815e45dc65"
server_key_version: "encryptionkey@0"
I20260923 19:34:24.235071 19790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:24.274673 19790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:24.279843 19790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:24.280141 19790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:24.281299 19790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:24.282917 19790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:24.283003 19790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:24.283128 19790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:24.283191 19790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:24.318056 19790 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33507
I20260923 19:34:24.318154 19917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33507 every 8 connection(s)
I20260923 19:34:24.319507 19790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:34:24.327111 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 19790
I20260923 19:34:24.327296 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:34:24.327739 388 external_mini_cluster.cc:1428] Setting key e422480f7391469c7d28e26bf54cde0d
Sep 23 19:34:24 dist-test-slave-kxfb krb5kdc[19684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:24.343330 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:24.322891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46389 (local address 127.0.97.62:45521)
0923 19:34:24.323771 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:24.323777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:24.324657 (+ 880us) server_negotiation.cc:407] Connection header received
0923 19:34:24.326868 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:24.326875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:24.326975 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:24.327091 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:24.329110 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:24.330094 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:24.331804 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:24.332246 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:24.332362 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:24.336522 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:24.336532 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:24.336537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:24.336587 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:24.339624 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:24.340616 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:24.340621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:24.340623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:24.340779 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:24.341271 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:24.341275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:24.341277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:24.341601 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:24.341816 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:24.342855 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0923 19:34:24.343046 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":584,"threads_started":1}
I20260923 19:34:24.344928 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:24.323554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45521 (local address 127.0.97.1:46389)
0923 19:34:24.324444 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:24.324504 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:24.326534 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:24.327425 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:24.327436 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:24.328106 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:24.328908 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:24.328928 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:24.330253 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:24.330257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:24.331655 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:24.331667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:24.332498 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:24.332505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:24.332759 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:24.333487 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:34:24.333516 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:24.336212 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:24.339787 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:24.339797 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:24.339820 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:24.340474 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:24.340886 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:24.340890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:24.340893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:24.341139 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:24.341829 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:24.341860 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:24.342401 (+ 541us) client_negotiation.cc:769] Sending connection context
0923 19:34:24.342735 (+ 334us) client_negotiation.cc:240] Negotiation successful
0923 19:34:24.343370 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":335,"threads_started":1}
I20260923 19:34:24.346895 19918 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45521
I20260923 19:34:24.347285 19918 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:24.348263 19918 heartbeater.cc:511] Master 127.0.97.62:45521 requested a full tablet report, sending...
I20260923 19:34:24.351068 19729 ts_manager.cc:194] Registered new tserver with Master: 85e592a47f7f43cd9f653d5adf428ddc (127.0.97.1:33507)
I20260923 19:34:24.352569 19729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46389
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 19:34:24.394501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45521
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:24.728993 19924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:24.729285 19924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:24.729391 19924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:24.736675 19924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:24.736832 19924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:24.736982 19924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:34:24.747051 19924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:24.748332 19924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:24.749583 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:24.763571 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:24.763631 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:24.763655 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:24.764465 19924 server_base.cc:1034] running on GCE node
I20260923 19:34:24.765093 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:24.766147 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:24.767390 19924 hybrid_clock.cc:648] HybridClock initialized: now 1790192064767325 us; error 62 us; skew 500 ppm
I20260923 19:34:24.767786 19924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:24 dist-test-slave-kxfb krb5kdc[19684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:24.771032 19924 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 19:34:24.772624 19924 webserver.cc:460] Webserver started at http://127.0.97.2:43535/ using document root <none> and password file <none>
I20260923 19:34:24.773291 19924 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:24.773396 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:24.773656 19924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:24.776582 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308"
format_stamp: "Formatted at 2026-09-23 19:34:24 on dist-test-slave-kxfb"
server_key: "2ac1018798d35b8adadf228c8db3ac71"
server_key_iv: "e53ff4d2b3486b311cfbd4d0064ea936"
server_key_version: "encryptionkey@0"
I20260923 19:34:24.777325 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308"
format_stamp: "Formatted at 2026-09-23 19:34:24 on dist-test-slave-kxfb"
server_key: "2ac1018798d35b8adadf228c8db3ac71"
server_key_iv: "e53ff4d2b3486b311cfbd4d0064ea936"
server_key_version: "encryptionkey@0"
I20260923 19:34:24.782964 19924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 19:34:24.786760 19939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:24.788328 19924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:34:24.788504 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308"
format_stamp: "Formatted at 2026-09-23 19:34:24 on dist-test-slave-kxfb"
server_key: "2ac1018798d35b8adadf228c8db3ac71"
server_key_iv: "e53ff4d2b3486b311cfbd4d0064ea936"
server_key_version: "encryptionkey@0"
I20260923 19:34:24.788743 19924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:24.800700 19924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:24.804920 19924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:24.805193 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:24.806165 19924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:24.807576 19924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:24.807658 19924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:24.807775 19924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:24.807838 19924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:24.842478 19924 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41299
I20260923 19:34:24.842579 20051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41299 every 8 connection(s)
I20260923 19:34:24.843884 19924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:34:24.852890 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 19924
I20260923 19:34:24.853071 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:34:24.853546 388 external_mini_cluster.cc:1428] Setting key 00eb2badb2f971a0f0f508a6a799865b
Sep 23 19:34:24 dist-test-slave-kxfb krb5kdc[19684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:24.866456 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:24.847067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53165 (local address 127.0.97.62:45521)
0923 19:34:24.847572 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:24.847577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:24.848573 (+ 996us) server_negotiation.cc:407] Connection header received
0923 19:34:24.850668 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:24.850674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:24.850767 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:24.850876 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:24.852901 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:24.853783 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:24.855586 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:24.856023 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:24.856123 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:24.860573 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:24.860578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:24.860581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:24.860615 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:24.863212 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:24.864059 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:24.864063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:24.864065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:24.864194 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:24.864633 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:24.864637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:24.864639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:24.864918 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:24.865050 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:24.865794 (+ 744us) server_negotiation.cc:299] Negotiation successful
0923 19:34:24.866038 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":233,"threads_started":1}
I20260923 19:34:24.867457 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:24.847553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45521 (local address 127.0.97.2:53165)
0923 19:34:24.848371 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:24.848422 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:24.850366 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:24.851160 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:24.851169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:24.851881 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:24.852655 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:24.852683 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:24.853953 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:24.853958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:24.855428 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:24.855451 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:24.856277 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:24.856285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:24.856578 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:24.857345 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:34:24.857377 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:24.860307 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:24.863388 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:24.863397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:24.863413 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:24.863923 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:24.864316 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:24.864321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:24.864325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:24.864510 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:24.865050 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:24.865060 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:24.865416 (+ 356us) client_negotiation.cc:769] Sending connection context
0923 19:34:24.865663 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 19:34:24.866244 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":344,"threads_started":1}
I20260923 19:34:24.868961 20052 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45521
I20260923 19:34:24.869447 20052 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:24.870525 20052 heartbeater.cc:511] Master 127.0.97.62:45521 requested a full tablet report, sending...
I20260923 19:34:24.872205 19729 ts_manager.cc:194] Registered new tserver with Master: fa6c3e1ff7f849f58b35b5a9bceb0308 (127.0.97.2:41299)
I20260923 19:34:24.873030 19729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53165
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 19:34:24.917965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45521
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:25.285512 20058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:25.285921 20058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:25.286154 20058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:25.295725 20058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:25.295855 20058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:25.295969 20058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:34:25.305825 20058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:25.307160 20058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:25.308400 20058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:25.321978 20063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:25.322280 20064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:25.323508 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:25.324937 20058 server_base.cc:1034] running on GCE node
I20260923 19:34:25.325616 20058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:25.326702 20058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:25.327947 20058 hybrid_clock.cc:648] HybridClock initialized: now 1790192065327880 us; error 55 us; skew 500 ppm
I20260923 19:34:25.328307 20058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:25 dist-test-slave-kxfb krb5kdc[19684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:25.331736 20058 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 19:34:25.333302 20058 webserver.cc:460] Webserver started at http://127.0.97.3:46051/ using document root <none> and password file <none>
I20260923 19:34:25.333998 20058 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:25.334112 20058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:25.334484 20058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:25.337342 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "18e713f5b6794b97abe2da0ef98d0155"
format_stamp: "Formatted at 2026-09-23 19:34:25 on dist-test-slave-kxfb"
server_key: "74f89db0b08c13bf6119990c66379672"
server_key_iv: "116e667a32c738e36d44183d8fb2a8de"
server_key_version: "encryptionkey@0"
I20260923 19:34:25.338095 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "18e713f5b6794b97abe2da0ef98d0155"
format_stamp: "Formatted at 2026-09-23 19:34:25 on dist-test-slave-kxfb"
server_key: "74f89db0b08c13bf6119990c66379672"
server_key_iv: "116e667a32c738e36d44183d8fb2a8de"
server_key_version: "encryptionkey@0"
I20260923 19:34:25.343629 20058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 19:34:25.347297 20073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:25.348709 20058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:34:25.348878 20058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "18e713f5b6794b97abe2da0ef98d0155"
format_stamp: "Formatted at 2026-09-23 19:34:25 on dist-test-slave-kxfb"
server_key: "74f89db0b08c13bf6119990c66379672"
server_key_iv: "116e667a32c738e36d44183d8fb2a8de"
server_key_version: "encryptionkey@0"
I20260923 19:34:25.349053 20058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:25.357034 19918 heartbeater.cc:503] Master 127.0.97.62:45521 was elected leader, sending a full tablet report...
I20260923 19:34:25.371093 20058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:25.375105 20058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:25.375346 20058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:25.376353 20058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:25.377744 20058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:25.377822 20058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:25.377919 20058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:25.377976 20058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:25.412340 20058 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43419
I20260923 19:34:25.412503 20185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43419 every 8 connection(s)
I20260923 19:34:25.413600 20058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:34:25.414892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 20058
I20260923 19:34:25.415124 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:34:25.415732 388 external_mini_cluster.cc:1428] Setting key 5ed2b79a9aa639954b33b3264c1dbc58
Sep 23 19:34:25 dist-test-slave-kxfb krb5kdc[19684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:25.436594 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.417028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39175 (local address 127.0.97.62:45521)
0923 19:34:25.417603 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.417610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.418752 (+ 1142us) server_negotiation.cc:407] Connection header received
0923 19:34:25.420918 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.420923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.421032 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.421132 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:25.422955 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.423843 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.425507 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.425938 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.426060 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.429983 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:25.429988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:25.429992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:25.430038 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:25.433348 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:25.434180 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:25.434184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:25.434187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:25.434373 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:25.434796 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:25.434799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:25.434801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:25.435098 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:25.435226 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.436102 (+ 876us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.436305 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":299,"threads_started":1}
I20260923 19:34:25.437909 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.417597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45521 (local address 127.0.97.3:39175)
0923 19:34:25.418528 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:25.418588 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:25.420617 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:25.421418 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:25.421428 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:25.422038 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:25.422774 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.422798 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.424005 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.424011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.425358 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.425371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.426175 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.426181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.426484 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.427235 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:34:25.427260 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:25.429723 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:25.433497 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:25.433530 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:25.433547 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:25.434041 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:25.434472 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:34:25.434475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:34:25.434478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:34:25.434688 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:34:25.435253 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:25.435260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:25.435687 (+ 427us) client_negotiation.cc:769] Sending connection context
0923 19:34:25.435969 (+ 282us) client_negotiation.cc:240] Negotiation successful
0923 19:34:25.436459 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":353,"threads_started":1}
I20260923 19:34:25.439684 20186 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45521
I20260923 19:34:25.440085 20186 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:25.441022 20186 heartbeater.cc:511] Master 127.0.97.62:45521 requested a full tablet report, sending...
I20260923 19:34:25.442960 19729 ts_manager.cc:194] Registered new tserver with Master: 18e713f5b6794b97abe2da0ef98d0155 (127.0.97.3:43419)
I20260923 19:34:25.443961 19729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39175
I20260923 19:34:25.445191 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:34:25.464371 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.449986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.0.97.62:45521)
0923 19:34:25.450191 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.450196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.450699 (+ 503us) server_negotiation.cc:407] Connection header received
0923 19:34:25.450835 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.450839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.450914 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.451028 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:25.452653 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.453504 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.454928 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.455336 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.455436 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.457949 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:25.457954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:25.457957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:25.457991 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:25.460992 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:25.462066 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:25.462072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:25.462077 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:25.462236 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:25.462680 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:25.462685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:25.462689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:25.463132 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:25.463395 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.463763 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.464027 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 19:34:25.468890 19729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:34:25.475956 19729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:34:25.522989 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.492207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.0.97.1:33507)
0923 19:34:25.513296 (+ 21089us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.513305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.513957 (+ 652us) server_negotiation.cc:407] Connection header received
0923 19:34:25.514168 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.514181 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.514622 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.514859 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:25.515827 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.517246 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.520328 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.521691 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.521810 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.522085 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.522401 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.522596 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20874,"thread_start_us":18469,"threads_started":1}
I20260923 19:34:25.523124 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.491496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33507 (local address 127.0.0.1:34736)
0923 19:34:25.513815 (+ 22319us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:25.513833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:25.513987 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:25.515179 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:25.515188 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:25.515219 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:25.515633 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.515647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.517408 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.517414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.520123 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.520137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.521888 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.521894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.522096 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.522151 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 19:34:25.522319 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 19:34:25.522560 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22185,"mutex_wait_us":37,"thread_start_us":22753,"threads_started":1}
I20260923 19:34:25.523910 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.492779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.0.97.3:43419)
0923 19:34:25.512923 (+ 20144us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.513024 (+ 101us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.514800 (+ 1776us) server_negotiation.cc:407] Connection header received
0923 19:34:25.514906 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.514912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.515166 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.515363 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:25.516683 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.518418 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.520720 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.521993 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.522126 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.523241 (+ 1115us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.523318 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.523491 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19881,"thread_start_us":19656,"threads_started":1}
I20260923 19:34:25.524384 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.491034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41299 (local address 127.0.0.1:53260)
0923 19:34:25.507013 (+ 15979us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:25.507077 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:25.507307 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:25.515725 (+ 8418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:25.515731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:25.515763 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:25.516415 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.516428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.518547 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.518552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.521059 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.521074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.523125 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.523132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.523703 (+ 571us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.523744 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 19:34:25.523841 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 19:34:25.523986 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15636,"mutex_wait_us":37,"thread_start_us":15974,"threads_started":1}
I20260923 19:34:25.524531 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.491264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.0.97.2:41299)
0923 19:34:25.515033 (+ 23769us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.515041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.515067 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:34:25.515155 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.515164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.515475 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.516608 (+ 1133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:25.516827 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.518380 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.521249 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.522636 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.522767 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.522933 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.523955 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.524172 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23603,"thread_start_us":23454,"threads_started":1}
I20260923 19:34:25.524461 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.491706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43419 (local address 127.0.0.1:44144)
0923 19:34:25.514531 (+ 22825us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:25.514547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:25.514671 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:25.515751 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:25.515756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:25.515780 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:25.516440 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.516453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.518596 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.518601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.520543 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.520557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.522476 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.522484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.522659 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.522698 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 19:34:25.522778 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 19:34:25.522949 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22682,"mutex_wait_us":28,"thread_start_us":21781,"threads_started":1}
I20260923 19:34:25.527904 19853 tablet_service.cc:1467] Processing CreateTablet for tablet 37310ef65da44c63a76de83f819bf8c3 (DEFAULT_TABLE table=test-table [id=19671ddebd824065b1767e947adddbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:25.528949 20121 tablet_service.cc:1467] Processing CreateTablet for tablet 37310ef65da44c63a76de83f819bf8c3 (DEFAULT_TABLE table=test-table [id=19671ddebd824065b1767e947adddbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:25.529526 19853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37310ef65da44c63a76de83f819bf8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:25.530403 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37310ef65da44c63a76de83f819bf8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:25.530350 19987 tablet_service.cc:1467] Processing CreateTablet for tablet 37310ef65da44c63a76de83f819bf8c3 (DEFAULT_TABLE table=test-table [id=19671ddebd824065b1767e947adddbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:25.531646 19987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37310ef65da44c63a76de83f819bf8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:25.543817 20205 tablet_bootstrap.cc:492] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: Bootstrap starting.
I20260923 19:34:25.546578 20206 tablet_bootstrap.cc:492] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: Bootstrap starting.
I20260923 19:34:25.548297 20207 tablet_bootstrap.cc:492] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: Bootstrap starting.
I20260923 19:34:25.548350 20205 tablet_bootstrap.cc:654] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:25.549813 20205 log.cc:824] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:25.551198 20206 tablet_bootstrap.cc:654] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:25.551977 20207 tablet_bootstrap.cc:654] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:25.552407 20206 log.cc:824] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:25.553176 20207 log.cc:824] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:25.553747 20205 tablet_bootstrap.cc:492] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: No bootstrap required, opened a new log
I20260923 19:34:25.554123 20205 ts_tablet_manager.cc:1397] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 19:34:25.555828 20206 tablet_bootstrap.cc:492] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: No bootstrap required, opened a new log
I20260923 19:34:25.556152 20206 ts_tablet_manager.cc:1397] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260923 19:34:25.556350 20207 tablet_bootstrap.cc:492] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: No bootstrap required, opened a new log
I20260923 19:34:25.556596 20207 ts_tablet_manager.cc:1397] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 19:34:25.562691 20205 raft_consensus.cc:348] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260923 19:34:25.563395 20205 raft_consensus.cc:374] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:25.563575 20205 raft_consensus.cc:729] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e713f5b6794b97abe2da0ef98d0155, State: Initialized, Role: FOLLOWER
I20260923 19:34:25.564363 20205 consensus_queue.cc:260] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260923 19:34:25.565593 20186 heartbeater.cc:503] Master 127.0.97.62:45521 was elected leader, sending a full tablet report...
I20260923 19:34:25.566519 20205 ts_tablet_manager.cc:1428] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260923 19:34:25.566932 20206 raft_consensus.cc:348] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260923 19:34:25.566932 20207 raft_consensus.cc:348] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260923 19:34:25.567440 20206 raft_consensus.cc:374] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:25.567440 20207 raft_consensus.cc:374] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:25.567541 20207 raft_consensus.cc:729] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6c3e1ff7f849f58b35b5a9bceb0308, State: Initialized, Role: FOLLOWER
I20260923 19:34:25.567587 20206 raft_consensus.cc:729] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e592a47f7f43cd9f653d5adf428ddc, State: Initialized, Role: FOLLOWER
I20260923 19:34:25.568246 20206 consensus_queue.cc:260] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260923 19:34:25.568243 20207 consensus_queue.cc:260] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260923 19:34:25.570143 20052 heartbeater.cc:503] Master 127.0.97.62:45521 was elected leader, sending a full tablet report...
I20260923 19:34:25.570637 20207 ts_tablet_manager.cc:1428] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260923 19:34:25.571955 20206 ts_tablet_manager.cc:1428] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260923 19:34:25.572527 19919 tablet.cc:2369] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 19:34:25.596040 20053 tablet.cc:2369] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:34:25.597901 20212 raft_consensus.cc:482] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:34:25.598315 20212 raft_consensus.cc:504] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260923 19:34:25.600601 20212 leader_election.cc:288] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6c3e1ff7f849f58b35b5a9bceb0308 (127.0.97.2:41299), 18e713f5b6794b97abe2da0ef98d0155 (127.0.97.3:43419)
I20260923 19:34:25.607554 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.601235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53739 (local address 127.0.97.2:41299)
0923 19:34:25.601684 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.601691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.602002 (+ 311us) server_negotiation.cc:407] Connection header received
0923 19:34:25.602221 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.602228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.602342 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.602427 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:25.603049 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.603773 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.605748 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.606636 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.606778 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.606873 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.607156 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.607284 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260923 19:34:25.607738 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.600877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41299 (local address 127.0.97.1:53739)
0923 19:34:25.601876 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:25.601892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:25.602069 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:25.602451 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:25.602455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:25.602482 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:25.602890 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.602910 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.603925 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.603930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.605580 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.605596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.606840 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.606845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.606994 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.607021 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 19:34:25.607116 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 19:34:25.607267 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":72,"thread_start_us":618,"threads_started":1}
I20260923 19:34:25.607856 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.601209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47349 (local address 127.0.97.3:43419)
0923 19:34:25.601455 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.601461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.602335 (+ 874us) server_negotiation.cc:407] Connection header received
0923 19:34:25.602552 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.602556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.602624 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.602702 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:25.603322 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.604142 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.605991 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.606916 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.607036 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.607151 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.607441 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.607593 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 19:34:25.608067 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.601163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43419 (local address 127.0.97.1:47349)
0923 19:34:25.602196 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:25.602214 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:25.602400 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:25.602753 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:25.602757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:25.602782 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:25.603169 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.603178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.604288 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.604292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.605834 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:25.605845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.607143 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.607149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:25.607285 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.607309 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 19:34:25.607391 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 19:34:25.607532 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":4,"thread_start_us":704,"threads_started":1}
I20260923 19:34:25.609175 20007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37310ef65da44c63a76de83f819bf8c3" candidate_uuid: "85e592a47f7f43cd9f653d5adf428ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" is_pre_election: true
I20260923 19:34:25.609206 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37310ef65da44c63a76de83f819bf8c3" candidate_uuid: "85e592a47f7f43cd9f653d5adf428ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e713f5b6794b97abe2da0ef98d0155" is_pre_election: true
I20260923 19:34:25.609761 20007 raft_consensus.cc:2454] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e592a47f7f43cd9f653d5adf428ddc in term 0.
I20260923 19:34:25.609797 20141 raft_consensus.cc:2454] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e592a47f7f43cd9f653d5adf428ddc in term 0.
I20260923 19:34:25.610498 19809 leader_election.cc:302] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e592a47f7f43cd9f653d5adf428ddc, fa6c3e1ff7f849f58b35b5a9bceb0308; no voters:
I20260923 19:34:25.610945 20212 raft_consensus.cc:2781] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:34:25.611102 20212 raft_consensus.cc:482] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:34:25.611207 20212 raft_consensus.cc:3037] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:25.613624 20212 raft_consensus.cc:504] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260923 19:34:25.614279 20212 leader_election.cc:288] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [CANDIDATE]: Term 1 election: Requested vote from peers fa6c3e1ff7f849f58b35b5a9bceb0308 (127.0.97.2:41299), 18e713f5b6794b97abe2da0ef98d0155 (127.0.97.3:43419)
I20260923 19:34:25.614856 20007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37310ef65da44c63a76de83f819bf8c3" candidate_uuid: "85e592a47f7f43cd9f653d5adf428ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308"
I20260923 19:34:25.614881 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37310ef65da44c63a76de83f819bf8c3" candidate_uuid: "85e592a47f7f43cd9f653d5adf428ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e713f5b6794b97abe2da0ef98d0155"
I20260923 19:34:25.615053 20141 raft_consensus.cc:3037] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:25.615073 20007 raft_consensus.cc:3037] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:25.617518 20007 raft_consensus.cc:2454] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e592a47f7f43cd9f653d5adf428ddc in term 1.
I20260923 19:34:25.617926 19809 leader_election.cc:302] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e592a47f7f43cd9f653d5adf428ddc, fa6c3e1ff7f849f58b35b5a9bceb0308; no voters:
I20260923 19:34:25.617981 20141 raft_consensus.cc:2454] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e592a47f7f43cd9f653d5adf428ddc in term 1.
I20260923 19:34:25.618232 20212 raft_consensus.cc:2781] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:25.618841 20212 raft_consensus.cc:686] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 1 LEADER]: Becoming Leader. State: Replica: 85e592a47f7f43cd9f653d5adf428ddc, State: Running, Role: LEADER
I20260923 19:34:25.619469 20212 consensus_queue.cc:237] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } }
I20260923 19:34:25.624486 19727 catalog_manager.cc:5495] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e592a47f7f43cd9f653d5adf428ddc (127.0.97.1). New cstate: current_term: 1 leader_uuid: "85e592a47f7f43cd9f653d5adf428ddc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85e592a47f7f43cd9f653d5adf428ddc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 } health_report { overall_health: UNKNOWN } } }
W20260923 19:34:25.665522 20187 tablet.cc:2369] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:34:25.667721 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.661826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.0.97.1:33507)
0923 19:34:25.662082 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.662087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.662139 (+ 52us) server_negotiation.cc:407] Connection header received
0923 19:34:25.662489 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.662493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.662580 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.662726 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:34:25.663075 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.663810 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.665268 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.665828 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.665964 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.666631 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:34:25.667139 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:34:25.667260 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.667398 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.667470 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 19:34:25.677819 20141 raft_consensus.cc:1261] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [term 1 FOLLOWER]: Refusing update from remote peer 85e592a47f7f43cd9f653d5adf428ddc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:25.678018 20007 raft_consensus.cc:1261] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [term 1 FOLLOWER]: Refusing update from remote peer 85e592a47f7f43cd9f653d5adf428ddc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:25.678858 20216 consensus_queue.cc:1035] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [LEADER]: Connected to new peer: Peer: permanent_uuid: "18e713f5b6794b97abe2da0ef98d0155" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:25.679390 20212 consensus_queue.cc:1035] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6c3e1ff7f849f58b35b5a9bceb0308" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:25.693262 20220 mvcc.cc:205] Tried to move back new op lower bound from 7332626701003984896 to 7332626700781522944. Current Snapshot: MvccSnapshot[applied={T|T < 7332626701003984896}]
I20260923 19:34:25.710099 19727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:25.710481 19727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:25.714679 19727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c7a0164f4274bfe8d45fcc6b3208498: Sending DeleteTablet for 3 replicas of tablet 37310ef65da44c63a76de83f819bf8c3
I20260923 19:34:25.716099 19987 tablet_service.cc:1514] Processing DeleteTablet for tablet 37310ef65da44c63a76de83f819bf8c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53260
I20260923 19:34:25.716144 19853 tablet_service.cc:1514] Processing DeleteTablet for tablet 37310ef65da44c63a76de83f819bf8c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34736
I20260923 19:34:25.716144 20121 tablet_service.cc:1514] Processing DeleteTablet for tablet 37310ef65da44c63a76de83f819bf8c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260923 19:34:25.718506 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 19790
I20260923 19:34:25.731559 20230 tablet_replica.cc:330] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: stopping tablet replica
I20260923 19:34:25.732198 20230 raft_consensus.cc:2229] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:25.732874 20230 raft_consensus.cc:2258] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:25.735419 20230 ts_tablet_manager.cc:1905] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:25.736150 20231 tablet_replica.cc:330] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: stopping tablet replica
I20260923 19:34:25.736339 20229 tablet_replica.cc:330] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: stopping tablet replica
I20260923 19:34:25.736955 20231 raft_consensus.cc:2229] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:34:25.737293 20229 raft_consensus.cc:2229] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:25.738013 20229 raft_consensus.cc:2258] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:25.738822 20231 raft_consensus.cc:2258] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 19:34:25 dist-test-slave-kxfb krb5kdc[19684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 19:34:25.741351 20229 ts_tablet_manager.cc:1905] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:25.745118 20230 ts_tablet_manager.cc:1918] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:25.745337 20230 log.cc:1196] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/37310ef65da44c63a76de83f819bf8c3
I20260923 19:34:25.745867 20230 ts_tablet_manager.cc:1939] T 37310ef65da44c63a76de83f819bf8c3 P 18e713f5b6794b97abe2da0ef98d0155: Deleting consensus metadata
I20260923 19:34:25.747730 20231 ts_tablet_manager.cc:1905] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:25.747853 19714 catalog_manager.cc:4841] TS 18e713f5b6794b97abe2da0ef98d0155 (127.0.97.3:43419): tablet 37310ef65da44c63a76de83f819bf8c3 (table test-table [id=19671ddebd824065b1767e947adddbc8]) successfully deleted
I20260923 19:34:25.750013 20229 ts_tablet_manager.cc:1918] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:25.750341 20229 log.cc:1196] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/37310ef65da44c63a76de83f819bf8c3
I20260923 19:34:25.750981 20229 ts_tablet_manager.cc:1939] T 37310ef65da44c63a76de83f819bf8c3 P fa6c3e1ff7f849f58b35b5a9bceb0308: Deleting consensus metadata
I20260923 19:34:25.752899 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.720959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.0.97.1:33507)
0923 19:34:25.721206 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.721214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.724587 (+ 3373us) server_negotiation.cc:407] Connection header received
0923 19:34:25.725543 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.725551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.725676 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.725770 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:25.731039 (+ 5269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.732520 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.734669 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.735556 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.735676 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.741990 (+ 6314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:25.742012 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:25.742039 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:25.742115 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:25.748144 (+ 6029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:25.749284 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:25.749294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:25.749297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:25.749527 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:25.750052 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:25.750058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:25.750061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:25.750587 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:25.750939 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.752161 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.752466 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 19:34:25.753352 19716 catalog_manager.cc:4841] TS fa6c3e1ff7f849f58b35b5a9bceb0308 (127.0.97.2:41299): tablet 37310ef65da44c63a76de83f819bf8c3 (table test-table [id=19671ddebd824065b1767e947adddbc8]) successfully deleted
I20260923 19:34:25.754200 19913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:25.863095 20231 ts_tablet_manager.cc:1918] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:25.863466 20231 log.cc:1196] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790191744267121-388-0/minicluster-data/ts-0/wal/wals/37310ef65da44c63a76de83f819bf8c3
I20260923 19:34:25.864068 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 19790
I20260923 19:34:25.864140 20231 ts_tablet_manager.cc:1939] T 37310ef65da44c63a76de83f819bf8c3 P 85e592a47f7f43cd9f653d5adf428ddc: Deleting consensus metadata
I20260923 19:34:25.881431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 19924
W20260923 19:34:25.881558 19713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33507 (error 108)
I20260923 19:34:25.883769 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.883160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33507 (local address 127.0.0.1:34766)
0923 19:34:25.883372 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:25.883530 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 19:34:25.884106 19713 catalog_manager.cc:4584] TS 85e592a47f7f43cd9f653d5adf428ddc (127.0.97.1:33507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37310ef65da44c63a76de83f819bf8c3: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33507: connect: Connection refused (error 111)
Sep 23 19:34:25 dist-test-slave-kxfb krb5kdc[19684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 19:34:25.897650 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.882356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.0.97.2:41299)
0923 19:34:25.882564 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:25.882570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:25.882638 (+ 68us) server_negotiation.cc:407] Connection header received
0923 19:34:25.882842 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:25.882846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:25.882925 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:25.883018 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:25.884611 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.885348 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.886677 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:25.887033 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:25.887140 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:25.891811 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:25.891824 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:25.891841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:25.891875 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:25.894929 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:25.895731 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:25.895739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:25.895742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:25.895861 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:25.896255 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:25.896259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:25.896261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:25.896664 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:25.896889 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:25.897226 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 19:34:25.897393 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 19:34:25.898550 20047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:25.938012 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.937441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33507 (local address 127.0.0.1:34778)
0923 19:34:25.937697 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:25.937804 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260923 19:34:25.939402 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.939055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33507 (local address 127.0.0.1:34786)
0923 19:34:25.939207 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:25.939295 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 19:34:25.991472 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 19924
I20260923 19:34:26.000320 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:25.999735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33507 (local address 127.0.0.1:34802)
0923 19:34:25.999976 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:26.000080 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 19:34:26.001797 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:26.001415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33507 (local address 127.0.0.1:34812)
0923 19:34:26.001594 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:26.001676 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 19:34:26.006790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 20058
Sep 23 19:34:26 dist-test-slave-kxfb krb5kdc[19684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 19:34:26.023921 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:26.007690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.0.97.3:43419)
0923 19:34:26.007895 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:26.007900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:26.007924 (+ 24us) server_negotiation.cc:407] Connection header received
0923 19:34:26.008085 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:26.008091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:26.008185 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:26.008299 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:26.009805 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:26.010685 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:26.012039 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:26.012456 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:26.012567 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:26.017541 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:26.017562 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:26.017584 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:26.017635 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:26.020663 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:26.021495 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:26.021504 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:26.021507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:26.021691 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:26.022065 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:26.022069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:26.022072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:26.022623 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:26.022938 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:26.023285 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 19:34:26.023541 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 19:34:26.024731 20181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:26.082701 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:26.081993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33507 (local address 127.0.0.1:34818)
0923 19:34:26.082327 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:26.082484 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260923 19:34:26.083969 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:26.083522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33507 (local address 127.0.0.1:34832)
0923 19:34:26.083775 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:26.083857 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260923 19:34:26.118474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 20058
I20260923 19:34:26.134858 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 19697
I20260923 19:34:26.136197 19759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:26.249327 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 19697
2026-09-23T19:34:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4409 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T19:34:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:26Z Disabled control of system clock
I20260923 19:34:26.333577 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39693
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 19:34:26.736675 20241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:26.737028 20241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:26.737116 20241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:26.744983 20241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:26.745121 20241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:26.745159 20241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:26.745219 20241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:26.745270 20241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:26.755636 20241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:26.757293 20241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:26.759455 20241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:26.773033 20246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:26.773092 20249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:26.773092 20247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:26.773816 20241 server_base.cc:1034] running on GCE node
I20260923 19:34:26.774863 20241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:26.776623 20241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:26.777842 20241 hybrid_clock.cc:648] HybridClock initialized: now 1790192066777773 us; error 70 us; skew 500 ppm
I20260923 19:34:26.778228 20241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:26.781011 20241 webserver.cc:460] Webserver started at http://127.0.97.62:45767/ using document root <none> and password file <none>
I20260923 19:34:26.781817 20241 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:26.781924 20241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:26.782265 20241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:26.785698 20241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "9768573c420d421a8b5b59134a885f5a"
format_stamp: "Formatted at 2026-09-23 19:34:26 on dist-test-slave-kxfb"
server_key: "f0bc8eda55279c4a8f10cce7cb529e12"
server_key_iv: "d2c9cbb188be87bad9cdcb937677bed2"
server_key_version: "encryptionkey@0"
I20260923 19:34:26.786616 20241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "9768573c420d421a8b5b59134a885f5a"
format_stamp: "Formatted at 2026-09-23 19:34:26 on dist-test-slave-kxfb"
server_key: "f0bc8eda55279c4a8f10cce7cb529e12"
server_key_iv: "d2c9cbb188be87bad9cdcb937677bed2"
server_key_version: "encryptionkey@0"
I20260923 19:34:26.792729 20241 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 19:34:26.797271 20255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:26.799037 20241 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 19:34:26.799260 20241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "9768573c420d421a8b5b59134a885f5a"
format_stamp: "Formatted at 2026-09-23 19:34:26 on dist-test-slave-kxfb"
server_key: "f0bc8eda55279c4a8f10cce7cb529e12"
server_key_iv: "d2c9cbb188be87bad9cdcb937677bed2"
server_key_version: "encryptionkey@0"
I20260923 19:34:26.799532 20241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:26.819792 20241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:26.820906 20241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:26.821241 20241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:26.842566 20241 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39693
I20260923 19:34:26.842608 20306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39693 every 8 connection(s)
I20260923 19:34:26.844226 20241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:26.847271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 20241
I20260923 19:34:26.847458 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:26.847972 388 external_mini_cluster.cc:1428] Setting key da96a4f07f0db660a53ae6cde178b438
I20260923 19:34:26.848521 20307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:26.863590 20307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a: Bootstrap starting.
I20260923 19:34:26.864260 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:26.850114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.0.97.62:39693)
0923 19:34:26.851271 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:26.851284 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:26.851341 (+ 57us) server_negotiation.cc:407] Connection header received
0923 19:34:26.853564 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:26.853589 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:26.854214 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:26.854669 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:26.855379 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:26.856896 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:26.858949 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:26.859730 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:26.859888 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:26.861297 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:26.861306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:26.861321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:26.861991 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:26.862058 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:26.862081 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:26.862173 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:26.862516 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 19:34:26.862960 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":400,"threads_started":1}
I20260923 19:34:26.867326 20307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:26.868516 20307 log.cc:824] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:26.871577 20307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a: No bootstrap required, opened a new log
I20260923 19:34:26.878893 20307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9768573c420d421a8b5b59134a885f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39693 } }
I20260923 19:34:26.879334 20307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:26.879467 20307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9768573c420d421a8b5b59134a885f5a, State: Initialized, Role: FOLLOWER
I20260923 19:34:26.880069 20307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9768573c420d421a8b5b59134a885f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39693 } }
I20260923 19:34:26.880296 20307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:26.880424 20307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:26.880569 20307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:26.882769 20307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9768573c420d421a8b5b59134a885f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39693 } }
I20260923 19:34:26.883282 20307 leader_election.cc:302] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9768573c420d421a8b5b59134a885f5a; no voters:
I20260923 19:34:26.883958 20307 leader_election.cc:288] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:26.884222 20312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:26.885363 20312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [term 1 LEADER]: Becoming Leader. State: Replica: 9768573c420d421a8b5b59134a885f5a, State: Running, Role: LEADER
I20260923 19:34:26.886008 20312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9768573c420d421a8b5b59134a885f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39693 } }
I20260923 19:34:26.886225 20307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:26.889912 20314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9768573c420d421a8b5b59134a885f5a. Latest consensus state: current_term: 1 leader_uuid: "9768573c420d421a8b5b59134a885f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9768573c420d421a8b5b59134a885f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39693 } } }
I20260923 19:34:26.890048 20313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9768573c420d421a8b5b59134a885f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9768573c420d421a8b5b59134a885f5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39693 } } }
I20260923 19:34:26.890475 20314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:26.890511 20313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:26.894955 20322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:26.899562 20322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:26.908242 20322 catalog_manager.cc:1347] Generated new cluster ID: ac28e8e18dda48e186f846d1cc4581e7
I20260923 19:34:26.908366 20322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:26.924047 20322 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:26.926203 20322 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:26.936128 20322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a: Generated new TSK 0
I20260923 19:34:26.937328 20322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:26.950798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39693
--builtin_ntp_servers=127.0.97.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 19:34:27.321545 20331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:27.321843 20331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:27.321929 20331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:27.329218 20331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:27.329344 20331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:27.329494 20331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 19:34:27.338711 20331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:27.340003 20331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:27.341763 20331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:27.355659 20336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:27.356021 20339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:27.356261 20337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:27.357184 20331 server_base.cc:1034] running on GCE node
I20260923 19:34:27.357821 20331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:27.358877 20331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:27.360121 20331 hybrid_clock.cc:648] HybridClock initialized: now 1790192067360056 us; error 55 us; skew 500 ppm
I20260923 19:34:27.360471 20331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:27.368769 20331 webserver.cc:460] Webserver started at http://127.0.97.1:42329/ using document root <none> and password file <none>
I20260923 19:34:27.369570 20331 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:27.369676 20331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:27.370045 20331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:27.373082 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/data/instance:
uuid: "6295651a3d734291bb08e3e3711524f1"
format_stamp: "Formatted at 2026-09-23 19:34:27 on dist-test-slave-kxfb"
server_key: "9a08ef4dc6abfaa3ed7235cf78b4e01f"
server_key_iv: "b5b28657db7752cd2014e958dfe0b4e8"
server_key_version: "encryptionkey@0"
I20260923 19:34:27.373875 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6295651a3d734291bb08e3e3711524f1"
format_stamp: "Formatted at 2026-09-23 19:34:27 on dist-test-slave-kxfb"
server_key: "9a08ef4dc6abfaa3ed7235cf78b4e01f"
server_key_iv: "b5b28657db7752cd2014e958dfe0b4e8"
server_key_version: "encryptionkey@0"
I20260923 19:34:27.379416 20331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 19:34:27.383301 20345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:27.384985 20331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:34:27.385165 20331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/wal
uuid: "6295651a3d734291bb08e3e3711524f1"
format_stamp: "Formatted at 2026-09-23 19:34:27 on dist-test-slave-kxfb"
server_key: "9a08ef4dc6abfaa3ed7235cf78b4e01f"
server_key_iv: "b5b28657db7752cd2014e958dfe0b4e8"
server_key_version: "encryptionkey@0"
I20260923 19:34:27.385387 20331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:27.403995 20331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:27.404806 20331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:27.405059 20331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:27.406261 20331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:27.407922 20331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:27.408023 20331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:27.408144 20331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:27.408217 20331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:27.443280 20331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36301
I20260923 19:34:27.443377 20457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36301 every 8 connection(s)
I20260923 19:34:27.444715 20331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/data/info.pb
I20260923 19:34:27.447656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 20331
I20260923 19:34:27.447866 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-0/wal/instance
I20260923 19:34:27.448453 388 external_mini_cluster.cc:1428] Setting key b022c567ec81d089c7581fe5529eca35
I20260923 19:34:27.451596 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39693
--builtin_ntp_servers=127.0.97.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 19:34:27.458400 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:27.448005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39917 (local address 127.0.97.62:39693)
0923 19:34:27.448805 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:27.448813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:27.450008 (+ 1195us) server_negotiation.cc:407] Connection header received
0923 19:34:27.451942 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:27.451948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:27.452050 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:27.452149 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:27.453301 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:27.454047 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:27.455644 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:27.456040 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:27.456149 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:27.457496 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:27.457501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:27.457504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:27.457524 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:27.457565 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:27.457576 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:27.457659 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:27.457887 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 19:34:27.457995 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":285,"threads_started":1}
I20260923 19:34:27.459380 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:27.448667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39693 (local address 127.0.97.1:39917)
0923 19:34:27.449807 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:27.449858 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:27.451583 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:27.452420 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:27.452429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:27.452523 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:27.453106 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:27.453134 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:27.454211 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:27.454215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:27.455486 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:27.455500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:27.456276 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:27.456284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:27.456495 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:27.457169 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:34:27.457171 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:27.457211 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:34:27.457298 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:27.457672 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:27.457684 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:27.457692 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 19:34:27.457848 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 19:34:27.458265 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":343,"threads_started":1}
I20260923 19:34:27.461208 20458 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39693
I20260923 19:34:27.461655 20458 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:27.462666 20458 heartbeater.cc:511] Master 127.0.97.62:39693 requested a full tablet report, sending...
I20260923 19:34:27.465255 20272 ts_manager.cc:194] Registered new tserver with Master: 6295651a3d734291bb08e3e3711524f1 (127.0.97.1:36301)
I20260923 19:34:27.467192 20272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39917
W20260923 19:34:27.789016 20462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:27.789345 20462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:27.789448 20462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:27.796793 20462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:27.796911 20462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:27.797050 20462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 19:34:27.806357 20462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:27.807628 20462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:27.809331 20462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:27.823438 20468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:27.823570 20467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:27.823462 20470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:27.824452 20462 server_base.cc:1034] running on GCE node
I20260923 19:34:27.825135 20462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:27.826217 20462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:27.827512 20462 hybrid_clock.cc:648] HybridClock initialized: now 1790192067827427 us; error 74 us; skew 500 ppm
I20260923 19:34:27.827901 20462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:27.830479 20462 webserver.cc:460] Webserver started at http://127.0.97.2:44843/ using document root <none> and password file <none>
I20260923 19:34:27.831189 20462 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:27.831290 20462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:27.831610 20462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:27.834492 20462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/data/instance:
uuid: "dea965291c484801bdf4efbb04f0c0e4"
format_stamp: "Formatted at 2026-09-23 19:34:27 on dist-test-slave-kxfb"
server_key: "b69035bf9f04b9fa9a3279bffd5ef18d"
server_key_iv: "4b51f3cedb0fb1c54da3cc3034da34ea"
server_key_version: "encryptionkey@0"
I20260923 19:34:27.835248 20462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dea965291c484801bdf4efbb04f0c0e4"
format_stamp: "Formatted at 2026-09-23 19:34:27 on dist-test-slave-kxfb"
server_key: "b69035bf9f04b9fa9a3279bffd5ef18d"
server_key_iv: "4b51f3cedb0fb1c54da3cc3034da34ea"
server_key_version: "encryptionkey@0"
I20260923 19:34:27.840757 20462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 19:34:27.844839 20476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:27.846578 20462 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 19:34:27.846786 20462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/wal
uuid: "dea965291c484801bdf4efbb04f0c0e4"
format_stamp: "Formatted at 2026-09-23 19:34:27 on dist-test-slave-kxfb"
server_key: "b69035bf9f04b9fa9a3279bffd5ef18d"
server_key_iv: "4b51f3cedb0fb1c54da3cc3034da34ea"
server_key_version: "encryptionkey@0"
I20260923 19:34:27.846999 20462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:27.867188 20462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:27.868031 20462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:27.868289 20462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:27.869335 20462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:27.870954 20462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:27.871053 20462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:27.871158 20462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:27.871228 20462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:27.905090 20462 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46175
I20260923 19:34:27.905200 20588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46175 every 8 connection(s)
I20260923 19:34:27.906461 20462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/data/info.pb
I20260923 19:34:27.912566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 20462
I20260923 19:34:27.912777 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/wal/instance
I20260923 19:34:27.913358 388 external_mini_cluster.cc:1428] Setting key 9cba1f95b52e93d0b0185395d774dba7
I20260923 19:34:27.916821 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39693
--builtin_ntp_servers=127.0.97.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 19:34:27.921016 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:27.909526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35269 (local address 127.0.97.62:39693)
0923 19:34:27.909728 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:27.909734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:27.911262 (+ 1528us) server_negotiation.cc:407] Connection header received
0923 19:34:27.913780 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:27.913786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:27.913874 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:27.913982 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:27.915404 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:27.916329 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:27.918056 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:27.918525 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:27.918640 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:27.920132 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:27.920136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:27.920140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:27.920160 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:27.920194 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:27.920207 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:27.920282 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:27.920537 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 19:34:27.920651 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 19:34:27.922477 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:27.909940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39693 (local address 127.0.97.2:35269)
0923 19:34:27.911053 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:27.911104 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:27.913375 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:27.914254 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:27.914263 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:27.914433 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:27.915215 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:27.915235 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:27.916489 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:27.916492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:27.917905 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:27.917917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:27.918759 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:27.918767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:27.919005 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:27.919786 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:34:27.919788 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:27.919835 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:34:27.919935 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:27.920297 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:27.920309 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:27.920317 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 19:34:27.920506 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 19:34:27.921009 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":361,"threads_started":1}
I20260923 19:34:27.924489 20589 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39693
I20260923 19:34:27.924876 20589 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:27.925730 20589 heartbeater.cc:511] Master 127.0.97.62:39693 requested a full tablet report, sending...
I20260923 19:34:27.927466 20272 ts_manager.cc:194] Registered new tserver with Master: dea965291c484801bdf4efbb04f0c0e4 (127.0.97.2:46175)
I20260923 19:34:27.928313 20272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35269
W20260923 19:34:28.263949 20592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:28.264317 20592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:28.264454 20592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:28.272737 20592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:28.272846 20592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:28.272992 20592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 19:34:28.282184 20592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:28.283542 20592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:28.285362 20592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:28.299492 20598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:28.299582 20600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:28.299602 20597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:28.300666 20592 server_base.cc:1034] running on GCE node
I20260923 19:34:28.301370 20592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:28.302521 20592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:28.303793 20592 hybrid_clock.cc:648] HybridClock initialized: now 1790192068303723 us; error 68 us; skew 500 ppm
I20260923 19:34:28.304262 20592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 19:34:28.307070 20592 webserver.cc:460] Webserver started at http://127.0.97.3:35349/ using document root <none> and password file <none>
I20260923 19:34:28.307914 20592 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:28.308023 20592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:28.308382 20592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:28.311833 20592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/data/instance:
uuid: "21bd711404d847099057d9127087b356"
format_stamp: "Formatted at 2026-09-23 19:34:28 on dist-test-slave-kxfb"
server_key: "ba4ac217f858ca8532a6326302f6c1a9"
server_key_iv: "26a3960abf003f40646fc147282761a9"
server_key_version: "encryptionkey@0"
I20260923 19:34:28.312709 20592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/wal/instance:
uuid: "21bd711404d847099057d9127087b356"
format_stamp: "Formatted at 2026-09-23 19:34:28 on dist-test-slave-kxfb"
server_key: "ba4ac217f858ca8532a6326302f6c1a9"
server_key_iv: "26a3960abf003f40646fc147282761a9"
server_key_version: "encryptionkey@0"
I20260923 19:34:28.318830 20592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 19:34:28.322965 20606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:28.324720 20592 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:34:28.324942 20592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/wal
uuid: "21bd711404d847099057d9127087b356"
format_stamp: "Formatted at 2026-09-23 19:34:28 on dist-test-slave-kxfb"
server_key: "ba4ac217f858ca8532a6326302f6c1a9"
server_key_iv: "26a3960abf003f40646fc147282761a9"
server_key_version: "encryptionkey@0"
I20260923 19:34:28.325187 20592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:28.351241 20592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:28.352432 20592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:28.352744 20592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:28.353984 20592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 19:34:28.355831 20592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 19:34:28.355954 20592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:28.356072 20592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 19:34:28.356148 20592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:28.391635 20592 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35131
I20260923 19:34:28.391718 20718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35131 every 8 connection(s)
I20260923 19:34:28.393132 20592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/data/info.pb
I20260923 19:34:28.395862 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 20592
I20260923 19:34:28.396067 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/wal/instance
I20260923 19:34:28.396661 388 external_mini_cluster.cc:1428] Setting key 9060e83dd272e0af188c184928dceb83
I20260923 19:34:28.407481 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.396280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49481 (local address 127.0.97.62:39693)
0923 19:34:28.396480 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.396484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.398510 (+ 2026us) server_negotiation.cc:407] Connection header received
0923 19:34:28.400834 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.400839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.400921 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.401005 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:28.402109 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.402915 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.404600 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.404991 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.405125 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.406608 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:28.406613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:28.406616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:28.406636 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:28.406671 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:28.406684 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:28.406761 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.407007 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.407117 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 19:34:28.408648 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.397072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39693 (local address 127.0.97.3:49481)
0923 19:34:28.398340 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.398388 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:28.400566 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:28.401254 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:28.401262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:28.401358 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:34:28.401942 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.401958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.403069 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.403073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.404434 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.404446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.405228 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.405235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.405455 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.406212 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 19:34:28.406215 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:34:28.406256 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 19:34:28.406381 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:34:28.406785 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:34:28.406796 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:34:28.406803 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 19:34:28.406967 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 19:34:28.407391 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":378,"threads_started":1}
I20260923 19:34:28.410526 20719 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39693
I20260923 19:34:28.411060 20719 heartbeater.cc:464] Registering TS with master...
I20260923 19:34:28.412256 20719 heartbeater.cc:511] Master 127.0.97.62:39693 requested a full tablet report, sending...
I20260923 19:34:28.414052 20272 ts_manager.cc:194] Registered new tserver with Master: 21bd711404d847099057d9127087b356 (127.0.97.3:35131)
I20260923 19:34:28.414942 20272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49481
I20260923 19:34:28.425884 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 19:34:28.436650 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.430572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.0.97.62:39693)
0923 19:34:28.430806 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.430811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.431206 (+ 395us) server_negotiation.cc:407] Connection header received
0923 19:34:28.431405 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.431410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.431495 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.431601 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:28.432096 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.432789 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.434430 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.434838 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.434932 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.436073 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:28.436078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:28.436081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:28.436099 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:28.436126 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:28.436136 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:28.436202 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.436330 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.436410 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 19:34:28.440894 20272 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 19:34:28.449939 20272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 19:34:28.470954 20458 heartbeater.cc:503] Master 127.0.97.62:39693 was elected leader, sending a full tablet report...
I20260923 19:34:28.476269 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.464807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35131 (local address 127.0.0.1:54380)
0923 19:34:28.466639 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.466663 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:28.466900 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:28.468475 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:28.468481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:28.468507 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:28.468854 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.468869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.470955 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.470961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.473376 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.473396 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.475296 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.475304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.475512 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.475567 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 19:34:28.475671 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 19:34:28.475888 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1672,"spinlock_wait_cycles":38912,"thread_start_us":4881,"threads_started":1}
I20260923 19:34:28.477428 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.465486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.0.97.3:35131)
0923 19:34:28.467873 (+ 2387us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.467878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.467907 (+ 29us) server_negotiation.cc:407] Connection header received
0923 19:34:28.467993 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.467998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.468242 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.469028 (+ 786us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:28.469302 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.470793 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.473632 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.475080 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.476519 (+ 1439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.476808 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.476889 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.477077 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2260,"thread_start_us":2118,"threads_started":1}
I20260923 19:34:28.477962 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.465169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36301 (local address 127.0.0.1:48752)
0923 19:34:28.469962 (+ 4793us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.469978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:28.470103 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:28.470657 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:28.470661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:28.470686 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:28.471231 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.471248 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.473111 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.473115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.474856 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.474868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.477187 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.477192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.477315 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.477341 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 19:34:28.477434 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 19:34:28.477575 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4678,"mutex_wait_us":26,"thread_start_us":875,"threads_started":1}
I20260923 19:34:28.478047 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.465978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.0.97.1:36301)
0923 19:34:28.467445 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.467456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.470116 (+ 2660us) server_negotiation.cc:407] Connection header received
0923 19:34:28.470262 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.470268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.470478 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.470655 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:28.471470 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.472968 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.475054 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.476276 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.476798 (+ 522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.477089 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.477507 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.477685 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":1194,"threads_started":1}
I20260923 19:34:28.478909 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.464769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46175 (local address 127.0.0.1:41054)
0923 19:34:28.466587 (+ 1818us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.466648 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:28.466903 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:28.469177 (+ 2274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:28.469182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:28.469206 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:28.469756 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.469770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.473587 (+ 3817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.473596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.475817 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.475831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.478086 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.478093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.478266 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.478329 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 19:34:28.478393 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 19:34:28.478528 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1446,"mutex_wait_us":4,"spinlock_wait_cycles":22912,"thread_start_us":354,"threads_started":1}
I20260923 19:34:28.479949 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.465495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.0.97.2:46175)
0923 19:34:28.468482 (+ 2987us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.468493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.468530 (+ 37us) server_negotiation.cc:407] Connection header received
0923 19:34:28.468649 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.468659 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.468935 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.469159 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:28.470020 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.473406 (+ 3386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.476474 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.477862 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.479272 (+ 1410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.479496 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.479563 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.479692 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2809,"thread_start_us":2688,"threads_started":1}
I20260923 19:34:28.482527 20654 tablet_service.cc:1467] Processing CreateTablet for tablet b8dc52c7f663437a9c5956d43eb76621 (DEFAULT_TABLE table=test-table [id=c7881192314041939e0e917639354dd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:28.482527 20393 tablet_service.cc:1467] Processing CreateTablet for tablet b8dc52c7f663437a9c5956d43eb76621 (DEFAULT_TABLE table=test-table [id=c7881192314041939e0e917639354dd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:28.484345 20393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8dc52c7f663437a9c5956d43eb76621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:28.484370 20654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8dc52c7f663437a9c5956d43eb76621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:28.487339 20524 tablet_service.cc:1467] Processing CreateTablet for tablet b8dc52c7f663437a9c5956d43eb76621 (DEFAULT_TABLE table=test-table [id=c7881192314041939e0e917639354dd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 19:34:28.488927 20524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8dc52c7f663437a9c5956d43eb76621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:28.500542 20737 tablet_bootstrap.cc:492] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: Bootstrap starting.
I20260923 19:34:28.502648 20738 tablet_bootstrap.cc:492] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1: Bootstrap starting.
I20260923 19:34:28.504534 20737 tablet_bootstrap.cc:654] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:28.505849 20737 log.cc:824] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:28.506191 20738 tablet_bootstrap.cc:654] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:28.507027 20739 tablet_bootstrap.cc:492] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: Bootstrap starting.
I20260923 19:34:28.507469 20738 log.cc:824] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:28.510690 20737 tablet_bootstrap.cc:492] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: No bootstrap required, opened a new log
I20260923 19:34:28.510936 20737 ts_tablet_manager.cc:1397] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 19:34:28.511049 20739 tablet_bootstrap.cc:654] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:28.511210 20738 tablet_bootstrap.cc:492] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1: No bootstrap required, opened a new log
I20260923 19:34:28.511538 20738 ts_tablet_manager.cc:1397] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260923 19:34:28.512472 20739 log.cc:824] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:28.516196 20739 tablet_bootstrap.cc:492] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: No bootstrap required, opened a new log
I20260923 19:34:28.516533 20739 ts_tablet_manager.cc:1397] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 19:34:28.518626 20737 raft_consensus.cc:348] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } }
I20260923 19:34:28.518996 20737 raft_consensus.cc:374] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:28.519102 20737 raft_consensus.cc:729] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea965291c484801bdf4efbb04f0c0e4, State: Initialized, Role: FOLLOWER
I20260923 19:34:28.519659 20737 consensus_queue.cc:260] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } }
I20260923 19:34:28.520844 20589 heartbeater.cc:503] Master 127.0.97.62:39693 was elected leader, sending a full tablet report...
I20260923 19:34:28.521267 20737 ts_tablet_manager.cc:1428] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260923 19:34:28.522104 20738 raft_consensus.cc:348] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } }
I20260923 19:34:28.522704 20738 raft_consensus.cc:374] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:28.522814 20738 raft_consensus.cc:729] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6295651a3d734291bb08e3e3711524f1, State: Initialized, Role: FOLLOWER
I20260923 19:34:28.523576 20738 consensus_queue.cc:260] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } }
I20260923 19:34:28.525616 20738 ts_tablet_manager.cc:1428] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260923 19:34:28.526875 20739 raft_consensus.cc:348] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } }
I20260923 19:34:28.527377 20739 raft_consensus.cc:374] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:28.527493 20739 raft_consensus.cc:729] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bd711404d847099057d9127087b356, State: Initialized, Role: FOLLOWER
I20260923 19:34:28.528225 20739 consensus_queue.cc:260] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } }
I20260923 19:34:28.529637 20719 heartbeater.cc:503] Master 127.0.97.62:39693 was elected leader, sending a full tablet report...
I20260923 19:34:28.530180 20739 ts_tablet_manager.cc:1428] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260923 19:34:28.530968 20743 raft_consensus.cc:482] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 19:34:28.531185 20743 raft_consensus.cc:504] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } }
I20260923 19:34:28.533067 20743 leader_election.cc:288] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21bd711404d847099057d9127087b356 (127.0.97.3:35131), 6295651a3d734291bb08e3e3711524f1 (127.0.97.1:36301)
I20260923 19:34:28.541666 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.533532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60731 (local address 127.0.97.1:36301)
0923 19:34:28.533724 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.533730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.534911 (+ 1181us) server_negotiation.cc:407] Connection header received
0923 19:34:28.535166 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.535175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.535304 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.535421 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:28.535952 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.537162 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.539384 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.540702 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.540859 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.540988 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.541311 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.541432 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 19:34:28.541769 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.533771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36301 (local address 127.0.97.2:60731)
0923 19:34:28.534792 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.534811 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:28.534963 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:28.535448 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:28.535451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:28.535473 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:28.535804 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.535814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.537310 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.537314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.539126 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.539138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.540928 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.540933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.541082 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.541119 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 19:34:28.541249 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 19:34:28.541393 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":534,"threads_started":1}
I20260923 19:34:28.543073 20413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8dc52c7f663437a9c5956d43eb76621" candidate_uuid: "dea965291c484801bdf4efbb04f0c0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6295651a3d734291bb08e3e3711524f1" is_pre_election: true
I20260923 19:34:28.543670 20413 raft_consensus.cc:2454] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea965291c484801bdf4efbb04f0c0e4 in term 0.
I20260923 19:34:28.544431 20480 leader_election.cc:302] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6295651a3d734291bb08e3e3711524f1, dea965291c484801bdf4efbb04f0c0e4; no voters:
I20260923 19:34:28.544787 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.533517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35003 (local address 127.0.97.3:35131)
0923 19:34:28.533731 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.533736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.538986 (+ 5250us) server_negotiation.cc:407] Connection header received
0923 19:34:28.539112 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.539117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.539194 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.539319 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 19:34:28.539836 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.540932 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.543028 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.543968 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.544086 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.544185 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.544447 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.544569 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 19:34:28.544839 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.534150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35131 (local address 127.0.97.2:35003)
0923 19:34:28.538832 (+ 4682us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.538853 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:34:28.538982 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:34:28.539336 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:34:28.539339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:34:28.539360 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 19:34:28.539662 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.539668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.541082 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.541086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.542820 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:34:28.542833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.544138 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.544145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:34:28.544296 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.544318 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 19:34:28.544399 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 19:34:28.544523 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4532,"thread_start_us":4339,"threads_started":1}
I20260923 19:34:28.544888 20743 raft_consensus.cc:2781] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 19:34:28.545042 20743 raft_consensus.cc:482] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 19:34:28.545151 20743 raft_consensus.cc:3037] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:28.546610 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8dc52c7f663437a9c5956d43eb76621" candidate_uuid: "dea965291c484801bdf4efbb04f0c0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bd711404d847099057d9127087b356" is_pre_election: true
I20260923 19:34:28.547093 20674 raft_consensus.cc:2454] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea965291c484801bdf4efbb04f0c0e4 in term 0.
I20260923 19:34:28.548156 20743 raft_consensus.cc:504] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } }
I20260923 19:34:28.548787 20743 leader_election.cc:288] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [CANDIDATE]: Term 1 election: Requested vote from peers 21bd711404d847099057d9127087b356 (127.0.97.3:35131), 6295651a3d734291bb08e3e3711524f1 (127.0.97.1:36301)
I20260923 19:34:28.549278 20674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8dc52c7f663437a9c5956d43eb76621" candidate_uuid: "dea965291c484801bdf4efbb04f0c0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bd711404d847099057d9127087b356"
I20260923 19:34:28.549456 20413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8dc52c7f663437a9c5956d43eb76621" candidate_uuid: "dea965291c484801bdf4efbb04f0c0e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6295651a3d734291bb08e3e3711524f1"
I20260923 19:34:28.549526 20674 raft_consensus.cc:3037] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:28.549693 20413 raft_consensus.cc:3037] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:28.553174 20413 raft_consensus.cc:2454] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea965291c484801bdf4efbb04f0c0e4 in term 1.
I20260923 19:34:28.553213 20674 raft_consensus.cc:2454] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea965291c484801bdf4efbb04f0c0e4 in term 1.
I20260923 19:34:28.553679 20480 leader_election.cc:302] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6295651a3d734291bb08e3e3711524f1, dea965291c484801bdf4efbb04f0c0e4; no voters:
I20260923 19:34:28.553992 20743 raft_consensus.cc:2781] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:28.554530 20743 raft_consensus.cc:686] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 1 LEADER]: Becoming Leader. State: Replica: dea965291c484801bdf4efbb04f0c0e4, State: Running, Role: LEADER
I20260923 19:34:28.555151 20743 consensus_queue.cc:237] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } }
I20260923 19:34:28.560405 20271 catalog_manager.cc:5495] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to dea965291c484801bdf4efbb04f0c0e4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "dea965291c484801bdf4efbb04f0c0e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea965291c484801bdf4efbb04f0c0e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 } health_report { overall_health: UNKNOWN } } }
I20260923 19:34:28.578802 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.573397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.0.97.2:46175)
0923 19:34:28.573656 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.573660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.573679 (+ 19us) server_negotiation.cc:407] Connection header received
0923 19:34:28.573772 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.573776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.573875 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.574005 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:34:28.574386 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.575132 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.576621 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.577044 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.577145 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.577658 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:34:28.578194 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:34:28.578349 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.578474 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.578582 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 19:34:28.589022 20413 raft_consensus.cc:1261] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [term 1 FOLLOWER]: Refusing update from remote peer dea965291c484801bdf4efbb04f0c0e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:28.589058 20674 raft_consensus.cc:1261] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [term 1 FOLLOWER]: Refusing update from remote peer dea965291c484801bdf4efbb04f0c0e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:34:28.590250 20743 consensus_queue.cc:1035] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21bd711404d847099057d9127087b356" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:28.590637 20748 consensus_queue.cc:1035] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6295651a3d734291bb08e3e3711524f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:34:28.603031 20751 mvcc.cc:205] Tried to move back new op lower bound from 7332626712927154176 to 7332626712806518784. Current Snapshot: MvccSnapshot[applied={T|T < 7332626712927154176}]
I20260923 19:34:28.606251 20752 mvcc.cc:205] Tried to move back new op lower bound from 7332626712927154176 to 7332626712806518784. Current Snapshot: MvccSnapshot[applied={T|T < 7332626712927154176}]
I20260923 19:34:28.619138 20271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:28.619462 20271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 19:34:28.623782 20271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9768573c420d421a8b5b59134a885f5a: Sending DeleteTablet for 3 replicas of tablet b8dc52c7f663437a9c5956d43eb76621
I20260923 19:34:28.625151 20524 tablet_service.cc:1514] Processing DeleteTablet for tablet b8dc52c7f663437a9c5956d43eb76621 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:28 UTC) from {username='slave'} at 127.0.0.1:41054
I20260923 19:34:28.625285 20654 tablet_service.cc:1514] Processing DeleteTablet for tablet b8dc52c7f663437a9c5956d43eb76621 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:28 UTC) from {username='slave'} at 127.0.0.1:54380
I20260923 19:34:28.625285 20393 tablet_service.cc:1514] Processing DeleteTablet for tablet b8dc52c7f663437a9c5956d43eb76621 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 19:34:28 UTC) from {username='slave'} at 127.0.0.1:48752
I20260923 19:34:28.628777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 20331
I20260923 19:34:28.637777 20761 tablet_replica.cc:330] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: stopping tablet replica
I20260923 19:34:28.638475 20761 raft_consensus.cc:2229] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:28.639194 20761 raft_consensus.cc:2258] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:28.641707 20762 tablet_replica.cc:330] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1: stopping tablet replica
I20260923 19:34:28.642268 20762 raft_consensus.cc:2229] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 19:34:28.642925 20762 raft_consensus.cc:2258] T b8dc52c7f663437a9c5956d43eb76621 P 6295651a3d734291bb08e3e3711524f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:28.646000 20761 ts_tablet_manager.cc:1905] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:28.652877 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.630234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.0.97.1:36301)
0923 19:34:28.630499 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.630507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.643788 (+ 13281us) server_negotiation.cc:407] Connection header received
0923 19:34:28.644356 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.644363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.644506 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.644591 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:28.645277 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.646542 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.648420 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.649023 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.649118 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.650816 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:28.650828 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:28.650850 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:28.651818 (+ 968us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:28.651894 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:28.651922 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:28.652021 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.652341 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.652484 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260923 19:34:28.653739 20761 ts_tablet_manager.cc:1918] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:28.653941 20761 log.cc:1196] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-2/wal/wals/b8dc52c7f663437a9c5956d43eb76621
I20260923 19:34:28.654575 20453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:28.655551 20760 tablet_replica.cc:330] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: stopping tablet replica
I20260923 19:34:28.655706 20761 ts_tablet_manager.cc:1939] T b8dc52c7f663437a9c5956d43eb76621 P 21bd711404d847099057d9127087b356: Deleting consensus metadata
I20260923 19:34:28.656296 20760 raft_consensus.cc:2229] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 1 LEADER]: Raft consensus shutting down.
I20260923 19:34:28.657267 20760 raft_consensus.cc:2258] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 19:34:28.657912 20256 catalog_manager.cc:4841] TS 21bd711404d847099057d9127087b356 (127.0.97.3:35131): tablet b8dc52c7f663437a9c5956d43eb76621 (table test-table [id=c7881192314041939e0e917639354dd6]) successfully deleted
I20260923 19:34:28.660879 20760 ts_tablet_manager.cc:1905] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 19:34:28.669231 20760 ts_tablet_manager.cc:1918] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 19:34:28.669397 20760 log.cc:1196] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: Deleting WAL directory at /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790191744267121-388-0/minicluster-data/ts-1/wal/wals/b8dc52c7f663437a9c5956d43eb76621
I20260923 19:34:28.669821 20760 ts_tablet_manager.cc:1939] T b8dc52c7f663437a9c5956d43eb76621 P dea965291c484801bdf4efbb04f0c0e4: Deleting consensus metadata
I20260923 19:34:28.671578 20257 catalog_manager.cc:4841] TS dea965291c484801bdf4efbb04f0c0e4 (127.0.97.2:46175): tablet b8dc52c7f663437a9c5956d43eb76621 (table test-table [id=c7881192314041939e0e917639354dd6]) successfully deleted
I20260923 19:34:28.780107 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 20331
I20260923 19:34:28.798369 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 20462
W20260923 19:34:28.798520 20259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36301 (error 108)
I20260923 19:34:28.801394 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.800799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36301 (local address 127.0.0.1:41270)
0923 19:34:28.801020 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.801182 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260923 19:34:28.801694 20259 catalog_manager.cc:4584] TS 6295651a3d734291bb08e3e3711524f1 (127.0.97.1:36301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8dc52c7f663437a9c5956d43eb76621: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36301: connect: Connection refused (error 111)
I20260923 19:34:28.806943 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.799195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.0.97.2:46175)
0923 19:34:28.799409 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.799415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.799509 (+ 94us) server_negotiation.cc:407] Connection header received
0923 19:34:28.799701 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.799707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.799851 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.799974 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:28.800619 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.801759 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.803424 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.803953 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.804128 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.805212 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:28.805225 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:28.805247 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:28.805995 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:28.806052 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:28.806075 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:28.806194 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.806405 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.806531 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 19:34:28.807812 20584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:28.855937 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.855304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36301 (local address 127.0.0.1:41282)
0923 19:34:28.855590 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.855730 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260923 19:34:28.857193 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.856834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36301 (local address 127.0.0.1:41284)
0923 19:34:28.857013 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.857096 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 19:34:28.917928 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.917414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36301 (local address 127.0.0.1:41300)
0923 19:34:28.917651 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.917737 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 19:34:28.918941 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.918607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36301 (local address 127.0.0.1:41304)
0923 19:34:28.918757 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:28.918838 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 19:34:28.937017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 20462
I20260923 19:34:28.955469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 20592
I20260923 19:34:28.962837 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.956288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.0.97.3:35131)
0923 19:34:28.956506 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:28.956513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:28.956555 (+ 42us) server_negotiation.cc:407] Connection header received
0923 19:34:28.956784 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:28.956789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:28.956928 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:28.957025 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:28.957462 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.958248 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.959568 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:28.960001 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:28.960112 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:28.961268 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:28.961277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:28.961296 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:28.961994 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:28.962049 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:28.962071 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:28.962173 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:28.962350 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 19:34:28.962477 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 19:34:28.963629 20714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:29.000552 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:28.999877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36301 (local address 127.0.0.1:41316)
0923 19:34:29.000215 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:29.000350 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260923 19:34:29.001814 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:29.001351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36301 (local address 127.0.0.1:41320)
0923 19:34:29.001600 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 19:34:29.001689 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260923 19:34:29.056866 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 20592
I20260923 19:34:29.073716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 20241
I20260923 19:34:29.075196 20302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:29.196169 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 20241
2026-09-23T19:34:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2952 ms)
[----------] 6 tests from EncryptionPolicyTest (22140 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:29 dist-test-slave-kxfb krb5kdc[20771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:29 dist-test-slave-kxfb krb5kdc[20771](info): set up 2 sockets
Sep 23 19:34:29 dist-test-slave-kxfb krb5kdc[20771](info): commencing operation
krb5kdc: starting...
W20260923 19:34:30.250813 388 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:30 dist-test-slave-kxfb krb5kdc[20771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:30.542541 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:30.886961 20784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:30.887249 20784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:30.887362 20784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:30.894753 20784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:30.894874 20784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:30.894924 20784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:30.895013 20784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:30.895084 20784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:30.904981 20784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33047
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:30.906312 20784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:30.907696 20784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:30.919806 20789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:30.919942 20792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:30.919986 20790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:30.921088 20784 server_base.cc:1034] running on GCE node
I20260923 19:34:30.922075 20784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:30.923913 20784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:30.925134 20784 hybrid_clock.cc:648] HybridClock initialized: now 1790192070925092 us; error 35 us; skew 500 ppm
I20260923 19:34:30.925539 20784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:30 dist-test-slave-kxfb krb5kdc[20771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:30.929153 20784 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:30.930930 20784 webserver.cc:460] Webserver started at http://127.0.0.1:43561/ using document root <none> and password file <none>
I20260923 19:34:30.931722 20784 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:30.931845 20784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:30.932190 20784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:30.935611 20784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b"
format_stamp: "Formatted at 2026-09-23 19:34:30 on dist-test-slave-kxfb"
server_key: "cbaf355211dd8e79732a20870a8ad7c3"
server_key_iv: "ad2f2058a6dd8169eedd16021a4655c6"
server_key_version: "encryptionkey@0"
I20260923 19:34:30.936429 20784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b"
format_stamp: "Formatted at 2026-09-23 19:34:30 on dist-test-slave-kxfb"
server_key: "cbaf355211dd8e79732a20870a8ad7c3"
server_key_iv: "ad2f2058a6dd8169eedd16021a4655c6"
server_key_version: "encryptionkey@0"
I20260923 19:34:30.942575 20784 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 19:34:30.946784 20799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:30.948455 20784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:34:30.948645 20784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b"
format_stamp: "Formatted at 2026-09-23 19:34:30 on dist-test-slave-kxfb"
server_key: "cbaf355211dd8e79732a20870a8ad7c3"
server_key_iv: "ad2f2058a6dd8169eedd16021a4655c6"
server_key_version: "encryptionkey@0"
I20260923 19:34:30.948900 20784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:30.967057 20784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:30.971944 20784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:30.972245 20784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:30.992473 20784 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33047
I20260923 19:34:30.992543 20850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33047 every 8 connection(s)
I20260923 19:34:30.994074 20784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:30.998198 20851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:30.998872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 20784
I20260923 19:34:30.999078 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:30.999745 388 external_mini_cluster.cc:1428] Setting key e1851f783bf7a45359000aad20a0fde9
I20260923 19:34:31.013127 20851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b: Bootstrap starting.
Sep 23 19:34:31 dist-test-slave-kxfb krb5kdc[20771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 19:34:31.017035 20851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:31.018251 20851 log.cc:824] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:31.022064 20851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b: No bootstrap required, opened a new log
I20260923 19:34:31.027426 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:31.002251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60053 (local address 127.0.0.1:33047)
0923 19:34:31.003236 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:31.003254 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:31.003306 (+ 52us) server_negotiation.cc:407] Connection header received
0923 19:34:31.005235 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:31.005261 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:31.005790 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:31.006209 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:31.008081 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:31.009558 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:31.011513 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:31.012327 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:31.012495 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:31.017825 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:31.017835 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:31.017855 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:31.017906 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:31.021931 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:31.023073 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:31.023081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:31.023089 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:31.023326 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:31.023890 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:31.023894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:31.023896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:31.024604 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:31.024917 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:31.025581 (+ 664us) server_negotiation.cc:299] Negotiation successful
0923 19:34:31.025993 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":421,"threads_started":1}
I20260923 19:34:31.031653 20851 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33047 } }
I20260923 19:34:31.032070 20851 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:31.032179 20851 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8c217ce8a94ee9b3ed2235a90c1e0b, State: Initialized, Role: FOLLOWER
I20260923 19:34:31.032541 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:31.033001 20851 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33047 } }
I20260923 19:34:31.033247 20851 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:31.033375 20851 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:31.033519 20851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:31.035998 20851 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33047 } }
I20260923 19:34:31.036578 20851 leader_election.cc:302] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd8c217ce8a94ee9b3ed2235a90c1e0b; no voters:
I20260923 19:34:31.037222 20851 leader_election.cc:288] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:31.037334 20860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:31.039719 20851 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:31.044278 20860 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [term 1 LEADER]: Becoming Leader. State: Replica: bd8c217ce8a94ee9b3ed2235a90c1e0b, State: Running, Role: LEADER
I20260923 19:34:31.045039 20860 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33047 } }
I20260923 19:34:31.049012 20861 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33047 } } }
I20260923 19:34:31.049067 20862 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd8c217ce8a94ee9b3ed2235a90c1e0b. Latest consensus state: current_term: 1 leader_uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8c217ce8a94ee9b3ed2235a90c1e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33047 } } }
I20260923 19:34:31.049660 20861 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:31.049682 20862 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:31.050853 20869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:31.061331 20869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:31.063465 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:31.038532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39837 (local address 127.0.0.1:33047)
0923 19:34:31.038785 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:31.038793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:31.043639 (+ 4846us) server_negotiation.cc:407] Connection header received
0923 19:34:31.043886 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:31.043896 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:31.043992 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:31.044087 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:31.046006 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:31.047111 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:31.049025 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:31.049819 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:31.049962 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:31.054486 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:31.054494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:31.054500 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:31.054550 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:31.058515 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:31.059692 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:31.059698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:31.059703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:31.059866 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:31.060448 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:31.060453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:31.060456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:31.060916 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:31.061217 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:31.062801 (+ 1584us) server_negotiation.cc:299] Negotiation successful
0923 19:34:31.063011 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 19:34:31.068670 20878 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 19:34:31.072616 20869 catalog_manager.cc:1347] Generated new cluster ID: f13c7e2abf1c40e980b7571e413492dc
I20260923 19:34:31.072747 20869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:31.103570 20869 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:31.105216 20869 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:31.112967 20869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd8c217ce8a94ee9b3ed2235a90c1e0b: Generated new TSK 0
I20260923 19:34:31.114220 20869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:31.129458 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 20784
I20260923 19:34:31.130957 20846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:31.232708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 20784
W20260923 19:34:31.251124 20769 connection.cc:535] client connection to 127.0.0.1:33047 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33047): Connection reset by peer (error 104)
2026-09-23T19:34:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:31 dist-test-slave-kxfb krb5kdc[20883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:31 dist-test-slave-kxfb krb5kdc[20883](info): set up 2 sockets
Sep 23 19:34:31 dist-test-slave-kxfb krb5kdc[20883](info): commencing operation
krb5kdc: starting...
W20260923 19:34:32.274729 388 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:32 dist-test-slave-kxfb krb5kdc[20883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:32.554129 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:32.881189 20896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:32.881479 20896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:32.881592 20896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:32.888720 20896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:32.888826 20896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:32.888873 20896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:32.888959 20896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:32.889021 20896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:32.898752 20896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:32.900009 20896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:32.901228 20896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:32.913141 20901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:32.913261 20902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:32.913347 20904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:32.914685 20896 server_base.cc:1034] running on GCE node
I20260923 19:34:32.915418 20896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:32.916805 20896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:32.918052 20896 hybrid_clock.cc:648] HybridClock initialized: now 1790192072917979 us; error 55 us; skew 500 ppm
I20260923 19:34:32.918458 20896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:32 dist-test-slave-kxfb krb5kdc[20883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:32.925397 20896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:32.927161 20896 webserver.cc:460] Webserver started at http://127.0.0.1:37141/ using document root <none> and password file <none>
I20260923 19:34:32.927871 20896 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:32.927977 20896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:32.928275 20896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:32.931391 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "8b9b827f48e846288734dfffc5dabf55"
format_stamp: "Formatted at 2026-09-23 19:34:32 on dist-test-slave-kxfb"
server_key: "af6910a29a26ee78c85e7811cf7dc6ea"
server_key_iv: "185fba1682592c8e61dae1f8cd4df50f"
server_key_version: "encryptionkey@0"
I20260923 19:34:32.932147 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "8b9b827f48e846288734dfffc5dabf55"
format_stamp: "Formatted at 2026-09-23 19:34:32 on dist-test-slave-kxfb"
server_key: "af6910a29a26ee78c85e7811cf7dc6ea"
server_key_iv: "185fba1682592c8e61dae1f8cd4df50f"
server_key_version: "encryptionkey@0"
I20260923 19:34:32.937649 20896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:34:32.941640 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:32.943233 20896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:34:32.943434 20896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "8b9b827f48e846288734dfffc5dabf55"
format_stamp: "Formatted at 2026-09-23 19:34:32 on dist-test-slave-kxfb"
server_key: "af6910a29a26ee78c85e7811cf7dc6ea"
server_key_iv: "185fba1682592c8e61dae1f8cd4df50f"
server_key_version: "encryptionkey@0"
I20260923 19:34:32.943617 20896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:32.964735 20896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:32.968725 20896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:32.968994 20896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:32.988081 20896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42635
I20260923 19:34:32.988157 20962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42635 every 8 connection(s)
I20260923 19:34:32.989419 20896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:32.992758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 20896
I20260923 19:34:32.993003 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:32.993590 388 external_mini_cluster.cc:1428] Setting key 85433a88b00cc452e274523be557ecc0
I20260923 19:34:32.994510 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:33.008653 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:32.995657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60799 (local address 127.0.0.1:42635)
0923 19:34:32.996508 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:32.996520 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:32.996565 (+ 45us) server_negotiation.cc:407] Connection header received
0923 19:34:32.998180 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:32.998200 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:32.998675 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:32.999051 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:32.999812 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:33.001314 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:33.003495 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:33.004260 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:33.004427 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:33.005960 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:33.005968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:33.005984 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:33.006676 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:33.006818 (+ 142us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:33.006846 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:33.006979 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:33.007310 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 19:34:33.007536 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":403,"threads_started":1}
I20260923 19:34:33.010535 20963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55: Bootstrap starting.
I20260923 19:34:33.012681 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:33.014779 20963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:33.016270 20963 log.cc:824] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:33.020143 20963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55: No bootstrap required, opened a new log
I20260923 19:34:33.025650 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:33.018498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60887 (local address 127.0.0.1:42635)
0923 19:34:33.018710 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:33.018717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:33.019112 (+ 395us) server_negotiation.cc:407] Connection header received
0923 19:34:33.019287 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:33.019293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:33.019380 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:33.019534 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:33.020104 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:33.021265 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:33.022692 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:33.023471 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:33.023630 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:33.024825 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:33.024832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:33.024837 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:33.024862 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:33.024915 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:33.024935 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:33.025061 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:33.025206 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 19:34:33.025273 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 19:34:33.030064 20963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9b827f48e846288734dfffc5dabf55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42635 } }
I20260923 19:34:33.030691 20963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:33.030803 20963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9b827f48e846288734dfffc5dabf55, State: Initialized, Role: FOLLOWER
I20260923 19:34:33.031627 20963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9b827f48e846288734dfffc5dabf55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42635 } }
I20260923 19:34:33.031961 20963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:33.032111 20963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:33.032310 20963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:33.036093 20963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9b827f48e846288734dfffc5dabf55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42635 } }
I20260923 19:34:33.036991 20963 leader_election.cc:302] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b9b827f48e846288734dfffc5dabf55; no voters:
I20260923 19:34:33.037850 20963 leader_election.cc:288] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:33.038113 20973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:33.039551 20973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9b827f48e846288734dfffc5dabf55, State: Running, Role: LEADER
I20260923 19:34:33.040470 20973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9b827f48e846288734dfffc5dabf55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42635 } }
I20260923 19:34:33.040741 20963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:33.044539 20974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b9b827f48e846288734dfffc5dabf55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9b827f48e846288734dfffc5dabf55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42635 } } }
I20260923 19:34:33.044625 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b9b827f48e846288734dfffc5dabf55. Latest consensus state: current_term: 1 leader_uuid: "8b9b827f48e846288734dfffc5dabf55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9b827f48e846288734dfffc5dabf55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42635 } } }
I20260923 19:34:33.045013 20974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:33.045097 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:33.052000 20981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:33.058888 20981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:33.072381 20981 catalog_manager.cc:1347] Generated new cluster ID: eb7523757cca4651a8a8a60e508f5c1e
I20260923 19:34:33.072496 20981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:33.087257 20981 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:33.088973 20981 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:33.099994 20981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b9b827f48e846288734dfffc5dabf55: Generated new TSK 0
I20260923 19:34:33.100888 20981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:33.108031 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 20896
W20260923 19:34:33.109586 20958 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60799
W20260923 19:34:33.110519 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 20896: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:33.110771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 20896
W20260923 19:34:33.128248 20881 connection.cc:535] client connection to 127.0.0.1:42635 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42635): Connection reset by peer (error 104)
2026-09-23T19:34:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:33 dist-test-slave-kxfb krb5kdc[20994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:33 dist-test-slave-kxfb krb5kdc[20994](info): set up 2 sockets
Sep 23 19:34:33 dist-test-slave-kxfb krb5kdc[20994](info): commencing operation
krb5kdc: starting...
W20260923 19:34:34.146667 388 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:34 dist-test-slave-kxfb krb5kdc[20994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:34.430464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:34.757418 21007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:34.757755 21007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:34.757891 21007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:34.765348 21007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:34.765499 21007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:34.765580 21007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:34.765676 21007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:34.765738 21007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:34.775993 21007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:34.777302 21007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:34.778683 21007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:34.790568 21012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:34.790925 21013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:34.790983 21015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:34.792485 21007 server_base.cc:1034] running on GCE node
I20260923 19:34:34.793295 21007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:34.794811 21007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:34.796068 21007 hybrid_clock.cc:648] HybridClock initialized: now 1790192074796009 us; error 60 us; skew 500 ppm
I20260923 19:34:34.796449 21007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:34 dist-test-slave-kxfb krb5kdc[20994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:34.799742 21007 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:34.801422 21007 webserver.cc:460] Webserver started at http://127.0.0.1:38301/ using document root <none> and password file <none>
I20260923 19:34:34.802110 21007 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:34.802228 21007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:34.802549 21007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:34.805541 21007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "9b8ba115988f4ff0a1afedfb647e1b8e"
format_stamp: "Formatted at 2026-09-23 19:34:34 on dist-test-slave-kxfb"
server_key: "f0fcf192213c6026c46e283fcef07677"
server_key_iv: "85c83f3a531a558efd5373df6666bb59"
server_key_version: "encryptionkey@0"
I20260923 19:34:34.806275 21007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b8ba115988f4ff0a1afedfb647e1b8e"
format_stamp: "Formatted at 2026-09-23 19:34:34 on dist-test-slave-kxfb"
server_key: "f0fcf192213c6026c46e283fcef07677"
server_key_iv: "85c83f3a531a558efd5373df6666bb59"
server_key_version: "encryptionkey@0"
I20260923 19:34:34.811698 21007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 19:34:34.815398 21022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:34.816946 21007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:34:34.817116 21007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "9b8ba115988f4ff0a1afedfb647e1b8e"
format_stamp: "Formatted at 2026-09-23 19:34:34 on dist-test-slave-kxfb"
server_key: "f0fcf192213c6026c46e283fcef07677"
server_key_iv: "85c83f3a531a558efd5373df6666bb59"
server_key_version: "encryptionkey@0"
I20260923 19:34:34.817297 21007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:34.822969 21007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:34.826890 21007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:34.827145 21007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:34.845883 21007 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40613
I20260923 19:34:34.846019 21073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40613 every 8 connection(s)
I20260923 19:34:34.847363 21007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:34.848325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 21007
I20260923 19:34:34.848523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:34.849011 388 external_mini_cluster.cc:1428] Setting key dad6dbb80b164a0cee440215e4da5c5d
I20260923 19:34:34.853152 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:34.859088 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:34.851102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44713 (local address 127.0.0.1:40613)
0923 19:34:34.852080 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:34.852099 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:34.852161 (+ 62us) server_negotiation.cc:407] Connection header received
0923 19:34:34.854744 (+ 2583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:34.854776 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:34.855262 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:34.855758 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:34.856857 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:34.856866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:34.856883 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:34.856900 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:34.856996 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:34.857024 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:34.857277 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:34.857420 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 19:34:34.857654 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":416,"threads_started":1}
I20260923 19:34:34.863101 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:34.867254 21074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e: Bootstrap starting.
I20260923 19:34:34.870433 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:34.867788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43317 (local address 127.0.0.1:40613)
0923 19:34:34.868004 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:34.868010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:34.868278 (+ 268us) server_negotiation.cc:407] Connection header received
0923 19:34:34.868513 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:34.868518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:34.868586 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:34.868676 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:34.869729 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:34.869733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:34.869736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:34.869751 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:34.869794 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:34.869814 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:34.869907 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:34.870031 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 19:34:34.870078 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 19:34:34.871718 21074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:34.872978 21074 log.cc:824] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:34.876173 21074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e: No bootstrap required, opened a new log
I20260923 19:34:34.884662 21074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ba115988f4ff0a1afedfb647e1b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260923 19:34:34.885118 21074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:34.885224 21074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8ba115988f4ff0a1afedfb647e1b8e, State: Initialized, Role: FOLLOWER
I20260923 19:34:34.885864 21074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ba115988f4ff0a1afedfb647e1b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260923 19:34:34.886102 21074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:34.886242 21074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:34.886444 21074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:34.888772 21074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ba115988f4ff0a1afedfb647e1b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260923 19:34:34.889348 21074 leader_election.cc:302] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b8ba115988f4ff0a1afedfb647e1b8e; no voters:
I20260923 19:34:34.890028 21074 leader_election.cc:288] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:34.890265 21084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:34.891409 21084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [term 1 LEADER]: Becoming Leader. State: Replica: 9b8ba115988f4ff0a1afedfb647e1b8e, State: Running, Role: LEADER
I20260923 19:34:34.892014 21084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ba115988f4ff0a1afedfb647e1b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } }
I20260923 19:34:34.892614 21074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:34.895339 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b8ba115988f4ff0a1afedfb647e1b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ba115988f4ff0a1afedfb647e1b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } } }
I20260923 19:34:34.895896 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:34.895455 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b8ba115988f4ff0a1afedfb647e1b8e. Latest consensus state: current_term: 1 leader_uuid: "9b8ba115988f4ff0a1afedfb647e1b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8ba115988f4ff0a1afedfb647e1b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40613 } } }
I20260923 19:34:34.896402 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:34.901814 21093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:34.907644 21093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:34.917863 21093 catalog_manager.cc:1347] Generated new cluster ID: 399cd16cbbc94936b2b0b5bc3dbd34c2
I20260923 19:34:34.918035 21093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:34.940893 21093 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:34.941521 21093 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:34.957646 21093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b8ba115988f4ff0a1afedfb647e1b8e: Generated new TSK 0
I20260923 19:34:34.958546 21093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:34.972231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21007
W20260923 19:34:34.973332 21069 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44713
W20260923 19:34:34.974269 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21007: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:34.974452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 21007
2026-09-23T19:34:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1860 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:35 dist-test-slave-kxfb krb5kdc[21105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:35 dist-test-slave-kxfb krb5kdc[21105](info): set up 2 sockets
Sep 23 19:34:35 dist-test-slave-kxfb krb5kdc[21105](info): commencing operation
krb5kdc: starting...
W20260923 19:34:36.006671 388 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:36 dist-test-slave-kxfb krb5kdc[21105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:36.290544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:36.620330 21118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:36.620609 21118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:36.620723 21118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:36.627838 21118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:36.627946 21118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:36.627996 21118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:36.628093 21118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:36.628149 21118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:36.637579 21118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:36.638798 21118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:36.640008 21118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:36.651620 21123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:36.651676 21124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:36.651700 21126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:36.653038 21118 server_base.cc:1034] running on GCE node
I20260923 19:34:36.653965 21118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:36.655643 21118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:36.656872 21118 hybrid_clock.cc:648] HybridClock initialized: now 1790192076656805 us; error 47 us; skew 500 ppm
I20260923 19:34:36.657259 21118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:36 dist-test-slave-kxfb krb5kdc[21105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:36.660647 21118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:36.662367 21118 webserver.cc:460] Webserver started at http://127.0.0.1:34025/ using document root <none> and password file <none>
I20260923 19:34:36.663416 21118 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:36.663563 21118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:36.663950 21118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:36.668349 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "99efa5050d3d48dea2f890c863e8546c"
format_stamp: "Formatted at 2026-09-23 19:34:36 on dist-test-slave-kxfb"
server_key: "ccec4a518d1e4f4a2d8dce10ff1c2b63"
server_key_iv: "b8a7744be08b039c52cd10ab7b7e01c1"
server_key_version: "encryptionkey@0"
I20260923 19:34:36.669298 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "99efa5050d3d48dea2f890c863e8546c"
format_stamp: "Formatted at 2026-09-23 19:34:36 on dist-test-slave-kxfb"
server_key: "ccec4a518d1e4f4a2d8dce10ff1c2b63"
server_key_iv: "b8a7744be08b039c52cd10ab7b7e01c1"
server_key_version: "encryptionkey@0"
I20260923 19:34:36.676071 21118 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 19:34:36.680850 21133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:36.683056 21118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 19:34:36.683312 21118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "99efa5050d3d48dea2f890c863e8546c"
format_stamp: "Formatted at 2026-09-23 19:34:36 on dist-test-slave-kxfb"
server_key: "ccec4a518d1e4f4a2d8dce10ff1c2b63"
server_key_iv: "b8a7744be08b039c52cd10ab7b7e01c1"
server_key_version: "encryptionkey@0"
I20260923 19:34:36.683604 21118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:36.714372 21118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:36.719180 21118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:36.719462 21118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:36.739697 21118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43413
I20260923 19:34:36.739722 21184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43413 every 8 connection(s)
I20260923 19:34:36.741336 21118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:36.745352 21185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:36.750559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 21118
I20260923 19:34:36.750767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:36.751232 388 external_mini_cluster.cc:1428] Setting key e6c6607ba734656007a7e43ad5360149
I20260923 19:34:36.757269 21185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c: Bootstrap starting.
I20260923 19:34:36.760866 21185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:36.761955 21185 log.cc:824] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:36.765211 21185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c: No bootstrap required, opened a new log
Sep 23 19:34:36 dist-test-slave-kxfb krb5kdc[21105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 19:34:36.774250 21185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99efa5050d3d48dea2f890c863e8546c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } }
I20260923 19:34:36.774713 21185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:36.774819 21185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99efa5050d3d48dea2f890c863e8546c, State: Initialized, Role: FOLLOWER
I20260923 19:34:36.775492 21185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99efa5050d3d48dea2f890c863e8546c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } }
I20260923 19:34:36.775821 21185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:36.775966 21185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:36.776120 21185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:36.778254 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:36.753549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44421 (local address 127.0.0.1:43413)
0923 19:34:36.754580 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:36.754597 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:36.754661 (+ 64us) server_negotiation.cc:407] Connection header received
0923 19:34:36.756970 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:36.756997 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:36.757511 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:36.757953 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:36.759501 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:36.761394 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:36.763048 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:36.763935 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:36.764125 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:36.768812 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:36.768823 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:36.768842 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:36.768891 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:36.773175 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:36.774091 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:36.774100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:36.774107 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:36.774365 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:36.774842 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:36.774848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:36.774851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:36.775562 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:36.775863 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:36.776435 (+ 572us) server_negotiation.cc:299] Negotiation successful
0923 19:34:36.776892 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":436,"threads_started":1}
I20260923 19:34:36.779320 21185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99efa5050d3d48dea2f890c863e8546c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } }
I20260923 19:34:36.779842 21185 leader_election.cc:302] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99efa5050d3d48dea2f890c863e8546c; no voters:
I20260923 19:34:36.780473 21185 leader_election.cc:288] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:36.780580 21190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:36.781656 21190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [term 1 LEADER]: Becoming Leader. State: Replica: 99efa5050d3d48dea2f890c863e8546c, State: Running, Role: LEADER
I20260923 19:34:36.782258 21190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99efa5050d3d48dea2f890c863e8546c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } }
I20260923 19:34:36.782878 21185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:36.783622 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:36.785197 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99efa5050d3d48dea2f890c863e8546c. Latest consensus state: current_term: 1 leader_uuid: "99efa5050d3d48dea2f890c863e8546c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99efa5050d3d48dea2f890c863e8546c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } } }
I20260923 19:34:36.785771 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:36.797107 21198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:36.796983 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99efa5050d3d48dea2f890c863e8546c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99efa5050d3d48dea2f890c863e8546c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43413 } } }
I20260923 19:34:36.797467 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:36.804961 21198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:36.809773 21212 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 19:34:36.814700 21198 catalog_manager.cc:1347] Generated new cluster ID: b7a0852f575e4db6bbd711909a8283be
I20260923 19:34:36.814821 21198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:36.820502 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:36.800627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57307 (local address 127.0.0.1:43413)
0923 19:34:36.800902 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:36.800910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:36.801908 (+ 998us) server_negotiation.cc:407] Connection header received
0923 19:34:36.802228 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:36.802236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:36.802396 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:36.802541 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:36.804494 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:36.805711 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:36.807965 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:36.808710 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:36.808836 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:36.812749 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:36.812758 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:36.812763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:36.812814 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:36.816795 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:36.817742 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:36.817748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:36.817753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:36.817914 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:36.818457 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:36.818462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:36.818466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:36.818924 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:36.819235 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:36.819588 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 19:34:36.819840 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260923 19:34:36.832878 21198 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:36.835011 21198 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:36.844753 21198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99efa5050d3d48dea2f890c863e8546c: Generated new TSK 0
I20260923 19:34:36.846163 21198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:36.855067 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21118
I20260923 19:34:36.856287 21180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:36.946808 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 21118
2026-09-23T19:34:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:37 dist-test-slave-kxfb krb5kdc[21217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:37 dist-test-slave-kxfb krb5kdc[21217](info): set up 2 sockets
Sep 23 19:34:37 dist-test-slave-kxfb krb5kdc[21217](info): commencing operation
krb5kdc: starting...
W20260923 19:34:37.982686 388 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:38 dist-test-slave-kxfb krb5kdc[21217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:38.265695 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:38.591606 21230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:38.591893 21230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:38.592024 21230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:38.599308 21230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:38.599422 21230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:38.599476 21230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:38.599565 21230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:38.599618 21230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:38.609318 21230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:38.610589 21230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:38.611827 21230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:38.624622 21235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:38.624609 21236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:38.624564 21238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:38.625844 21230 server_base.cc:1034] running on GCE node
I20260923 19:34:38.626861 21230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:38.628439 21230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:38.629652 21230 hybrid_clock.cc:648] HybridClock initialized: now 1790192078629591 us; error 56 us; skew 500 ppm
I20260923 19:34:38.630029 21230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:38 dist-test-slave-kxfb krb5kdc[21217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:38.636806 21230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:38.638552 21230 webserver.cc:460] Webserver started at http://127.0.0.1:46207/ using document root <none> and password file <none>
I20260923 19:34:38.639245 21230 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:38.639349 21230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:38.639647 21230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:38.642702 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "d58f3aa969284947a9624435c0662d6c"
format_stamp: "Formatted at 2026-09-23 19:34:38 on dist-test-slave-kxfb"
server_key: "9645d36b5c7723bddd24fcf8ac1eabd0"
server_key_iv: "f49450db8060ab6958953ada746381dd"
server_key_version: "encryptionkey@0"
I20260923 19:34:38.643482 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "d58f3aa969284947a9624435c0662d6c"
format_stamp: "Formatted at 2026-09-23 19:34:38 on dist-test-slave-kxfb"
server_key: "9645d36b5c7723bddd24fcf8ac1eabd0"
server_key_iv: "f49450db8060ab6958953ada746381dd"
server_key_version: "encryptionkey@0"
I20260923 19:34:38.649557 21230 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 19:34:38.653505 21245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:38.654929 21230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:34:38.655122 21230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "d58f3aa969284947a9624435c0662d6c"
format_stamp: "Formatted at 2026-09-23 19:34:38 on dist-test-slave-kxfb"
server_key: "9645d36b5c7723bddd24fcf8ac1eabd0"
server_key_iv: "f49450db8060ab6958953ada746381dd"
server_key_version: "encryptionkey@0"
I20260923 19:34:38.655303 21230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:38.678062 21230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:38.682325 21230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:38.682605 21230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:38.701859 21230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44333
I20260923 19:34:38.701956 21296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44333 every 8 connection(s)
I20260923 19:34:38.703299 21230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:38.704401 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 21230
I20260923 19:34:38.704586 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:38.705026 388 external_mini_cluster.cc:1428] Setting key bc6ff941765d0997f70ed6d2863481fa
I20260923 19:34:38.708913 21297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:38.722332 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:38.707117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55741 (local address 127.0.0.1:44333)
0923 19:34:38.708106 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:38.708125 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:38.708191 (+ 66us) server_negotiation.cc:407] Connection header received
0923 19:34:38.710836 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:38.710872 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:38.711490 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:38.712102 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:38.712717 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:38.714852 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:38.716326 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:38.717464 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:38.717675 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:38.718862 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:38.718875 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:38.718898 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:38.719727 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:38.719871 (+ 144us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:38.719909 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:38.720053 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:38.720300 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 19:34:38.720670 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":395,"threads_started":1}
I20260923 19:34:38.724548 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c: Bootstrap starting.
I20260923 19:34:38.726773 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:38.728359 21297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:38.729494 21297 log.cc:824] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:38.733487 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c: No bootstrap required, opened a new log
I20260923 19:34:38.738334 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:38.731683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33717 (local address 127.0.0.1:44333)
0923 19:34:38.731925 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:38.731931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:38.732152 (+ 221us) server_negotiation.cc:407] Connection header received
0923 19:34:38.732326 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:38.732332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:38.732424 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:38.732577 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:38.733097 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:38.734213 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:38.735682 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:38.736379 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:38.736527 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:38.737510 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:38.737517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:38.737523 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:38.737548 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:38.737597 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:38.737618 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:38.737726 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:38.737853 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 19:34:38.737970 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 19:34:38.743068 21297 raft_consensus.cc:348] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58f3aa969284947a9624435c0662d6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260923 19:34:38.743539 21297 raft_consensus.cc:374] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:38.743618 21297 raft_consensus.cc:729] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58f3aa969284947a9624435c0662d6c, State: Initialized, Role: FOLLOWER
I20260923 19:34:38.744256 21297 consensus_queue.cc:260] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58f3aa969284947a9624435c0662d6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260923 19:34:38.744496 21297 raft_consensus.cc:388] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:38.744604 21297 raft_consensus.cc:482] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:38.744748 21297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:38.747458 21297 raft_consensus.cc:504] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58f3aa969284947a9624435c0662d6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260923 19:34:38.748009 21297 leader_election.cc:302] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d58f3aa969284947a9624435c0662d6c; no voters:
I20260923 19:34:38.748670 21297 leader_election.cc:288] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:38.748867 21307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:38.749699 21307 raft_consensus.cc:686] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [term 1 LEADER]: Becoming Leader. State: Replica: d58f3aa969284947a9624435c0662d6c, State: Running, Role: LEADER
I20260923 19:34:38.750384 21307 consensus_queue.cc:237] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58f3aa969284947a9624435c0662d6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } }
I20260923 19:34:38.751132 21297 sys_catalog.cc:564] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:38.754057 21308 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d58f3aa969284947a9624435c0662d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58f3aa969284947a9624435c0662d6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } } }
I20260923 19:34:38.754581 21308 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:38.754004 21309 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d58f3aa969284947a9624435c0662d6c. Latest consensus state: current_term: 1 leader_uuid: "d58f3aa969284947a9624435c0662d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58f3aa969284947a9624435c0662d6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44333 } } }
I20260923 19:34:38.754971 21309 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:38.756333 21316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:38.761665 21316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:38.772712 21316 catalog_manager.cc:1347] Generated new cluster ID: db39ce5489224aa081eb0e97496e58d1
I20260923 19:34:38.772843 21316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:38.796640 21316 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:38.798367 21316 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:38.809567 21316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d58f3aa969284947a9624435c0662d6c: Generated new TSK 0
I20260923 19:34:38.810684 21316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:38.818921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21230
W20260923 19:34:38.820132 21292 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55741
W20260923 19:34:38.820901 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21230: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:38.821072 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 21230
2026-09-23T19:34:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:38 dist-test-slave-kxfb krb5kdc[21328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:38 dist-test-slave-kxfb krb5kdc[21328](info): set up 2 sockets
Sep 23 19:34:38 dist-test-slave-kxfb krb5kdc[21328](info): commencing operation
krb5kdc: starting...
W20260923 19:34:39.854534 388 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:40 dist-test-slave-kxfb krb5kdc[21328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:40.137382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:40.463357 21341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:40.463646 21341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:40.463738 21341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:40.470858 21341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:40.470986 21341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:40.471033 21341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:40.471122 21341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:40.471158 21341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:40.481206 21341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:40.482520 21341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:40.483749 21341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:40.496361 21346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:40.496447 21349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:40.496397 21347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:40.499934 21341 server_base.cc:1034] running on GCE node
I20260923 19:34:40.500771 21341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:40.502245 21341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:40.503508 21341 hybrid_clock.cc:648] HybridClock initialized: now 1790192080503448 us; error 56 us; skew 500 ppm
I20260923 19:34:40.503840 21341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:40 dist-test-slave-kxfb krb5kdc[21328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:40.507076 21341 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:40.508715 21341 webserver.cc:460] Webserver started at http://127.0.0.1:38383/ using document root <none> and password file <none>
I20260923 19:34:40.509399 21341 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:40.509514 21341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:40.509796 21341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:40.512850 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "830c6e53c8374aff91ae3d6d6c6e9962"
format_stamp: "Formatted at 2026-09-23 19:34:40 on dist-test-slave-kxfb"
server_key: "be77795ae14720c98d096e89d7e5ac7d"
server_key_iv: "18dbd6368e8319fe7b5b29760dbf6c59"
server_key_version: "encryptionkey@0"
I20260923 19:34:40.513608 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "830c6e53c8374aff91ae3d6d6c6e9962"
format_stamp: "Formatted at 2026-09-23 19:34:40 on dist-test-slave-kxfb"
server_key: "be77795ae14720c98d096e89d7e5ac7d"
server_key_iv: "18dbd6368e8319fe7b5b29760dbf6c59"
server_key_version: "encryptionkey@0"
I20260923 19:34:40.519208 21341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 19:34:40.523082 21356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:40.524715 21341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:34:40.524921 21341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "830c6e53c8374aff91ae3d6d6c6e9962"
format_stamp: "Formatted at 2026-09-23 19:34:40 on dist-test-slave-kxfb"
server_key: "be77795ae14720c98d096e89d7e5ac7d"
server_key_iv: "18dbd6368e8319fe7b5b29760dbf6c59"
server_key_version: "encryptionkey@0"
I20260923 19:34:40.525117 21341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:40.531023 21341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:40.535004 21341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:40.535281 21341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:40.554158 21341 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34057
I20260923 19:34:40.554242 21407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34057 every 8 connection(s)
I20260923 19:34:40.555534 21341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:40.560330 21408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:40.565379 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 21341
I20260923 19:34:40.565605 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:40.566212 388 external_mini_cluster.cc:1428] Setting key 945d5370cb6d0ae3a72344a3fdcf8657
I20260923 19:34:40.573807 21408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962: Bootstrap starting.
I20260923 19:34:40.576107 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:40.568438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34609 (local address 127.0.0.1:34057)
0923 19:34:40.569578 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:40.569595 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:40.570038 (+ 443us) server_negotiation.cc:407] Connection header received
0923 19:34:40.572443 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:40.572469 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:40.572863 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:40.573223 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:40.574450 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:40.574457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:40.574470 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:40.574492 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:40.574567 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:40.574589 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:40.574787 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:40.574894 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 19:34:40.575088 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":478,"threads_started":1}
I20260923 19:34:40.577790 21408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:40.579174 21408 log.cc:824] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:40.580077 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:40.583045 21408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962: No bootstrap required, opened a new log
I20260923 19:34:40.588194 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:40.585678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53645 (local address 127.0.0.1:34057)
0923 19:34:40.585929 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:40.585940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:40.586089 (+ 149us) server_negotiation.cc:407] Connection header received
0923 19:34:40.586281 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:40.586310 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:40.586388 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:40.586536 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:40.587486 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:40.587492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:40.587496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:40.587515 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:40.587567 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:40.587586 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:40.587732 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:40.587820 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 19:34:40.587889 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 19:34:40.592687 21408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830c6e53c8374aff91ae3d6d6c6e9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260923 19:34:40.593129 21408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:40.593250 21408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830c6e53c8374aff91ae3d6d6c6e9962, State: Initialized, Role: FOLLOWER
I20260923 19:34:40.593894 21408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830c6e53c8374aff91ae3d6d6c6e9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260923 19:34:40.594116 21408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:40.594262 21408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:40.594458 21408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:40.596980 21408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830c6e53c8374aff91ae3d6d6c6e9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260923 19:34:40.597550 21408 leader_election.cc:302] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 830c6e53c8374aff91ae3d6d6c6e9962; no voters:
I20260923 19:34:40.598248 21408 leader_election.cc:288] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:40.598464 21418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:40.599225 21418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [term 1 LEADER]: Becoming Leader. State: Replica: 830c6e53c8374aff91ae3d6d6c6e9962, State: Running, Role: LEADER
I20260923 19:34:40.599820 21418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830c6e53c8374aff91ae3d6d6c6e9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260923 19:34:40.600620 21408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:40.603608 21419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "830c6e53c8374aff91ae3d6d6c6e9962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830c6e53c8374aff91ae3d6d6c6e9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } } }
I20260923 19:34:40.603648 21420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 830c6e53c8374aff91ae3d6d6c6e9962. Latest consensus state: current_term: 1 leader_uuid: "830c6e53c8374aff91ae3d6d6c6e9962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830c6e53c8374aff91ae3d6d6c6e9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } } }
I20260923 19:34:40.604028 21419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:40.604051 21420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:40.606356 21425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:40.611347 21425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:40.619632 21425 catalog_manager.cc:1347] Generated new cluster ID: e3be0c1bc9a54279b3495e6f3bfbec10
I20260923 19:34:40.619761 21425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:40.643051 21425 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:40.643481 21425 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:40.653402 21425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 830c6e53c8374aff91ae3d6d6c6e9962: Generated new TSK 0
I20260923 19:34:40.654174 21425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:40.674552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21341
W20260923 19:34:40.676335 21403 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34609
W20260923 19:34:40.677250 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21341: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:40.677438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 21341
2026-09-23T19:34:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1854 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:40 dist-test-slave-kxfb krb5kdc[21439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:40 dist-test-slave-kxfb krb5kdc[21439](info): set up 2 sockets
Sep 23 19:34:40 dist-test-slave-kxfb krb5kdc[21439](info): commencing operation
krb5kdc: starting...
W20260923 19:34:41.726691 388 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:41 dist-test-slave-kxfb krb5kdc[21439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:42.011861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:42.340631 21452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:42.340941 21452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:42.341048 21452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:42.348919 21452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:42.349045 21452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:42.349092 21452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:42.349180 21452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:42.349244 21452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:42.359480 21452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:42.360806 21452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:42.362064 21452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:42.374562 21457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:42.374734 21458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:42.374619 21460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:42.375914 21452 server_base.cc:1034] running on GCE node
I20260923 19:34:42.376739 21452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:42.378278 21452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:42.379587 21452 hybrid_clock.cc:648] HybridClock initialized: now 1790192082379498 us; error 67 us; skew 500 ppm
I20260923 19:34:42.379978 21452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:42 dist-test-slave-kxfb krb5kdc[21439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:42.383319 21452 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:42.384959 21452 webserver.cc:460] Webserver started at http://127.0.0.1:45391/ using document root <none> and password file <none>
I20260923 19:34:42.385654 21452 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:42.385766 21452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:42.386039 21452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:42.389091 21452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "2c8a80320b4f4d159cea1ac628f27852"
format_stamp: "Formatted at 2026-09-23 19:34:42 on dist-test-slave-kxfb"
server_key: "5948c704c131b6297d9eef96da608544"
server_key_iv: "2e60a883e1793b562370f9aefe23394c"
server_key_version: "encryptionkey@0"
I20260923 19:34:42.389851 21452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "2c8a80320b4f4d159cea1ac628f27852"
format_stamp: "Formatted at 2026-09-23 19:34:42 on dist-test-slave-kxfb"
server_key: "5948c704c131b6297d9eef96da608544"
server_key_iv: "2e60a883e1793b562370f9aefe23394c"
server_key_version: "encryptionkey@0"
I20260923 19:34:42.395637 21452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260923 19:34:42.399773 21467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:42.401419 21452 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:34:42.401623 21452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "2c8a80320b4f4d159cea1ac628f27852"
format_stamp: "Formatted at 2026-09-23 19:34:42 on dist-test-slave-kxfb"
server_key: "5948c704c131b6297d9eef96da608544"
server_key_iv: "2e60a883e1793b562370f9aefe23394c"
server_key_version: "encryptionkey@0"
I20260923 19:34:42.401829 21452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:42.427143 21452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:42.431849 21452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:42.432111 21452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:42.451627 21452 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44461
I20260923 19:34:42.451725 21518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44461 every 8 connection(s)
I20260923 19:34:42.453009 21452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:42.457401 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 21452
I20260923 19:34:42.457564 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:42.457671 21519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:42.458052 388 external_mini_cluster.cc:1428] Setting key 7362ed2eeb1b9c0357b4c5bcf04aaf6e
Sep 23 19:34:42 dist-test-slave-kxfb krb5kdc[21439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 19:34:42.473923 21519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852: Bootstrap starting.
I20260923 19:34:42.478607 21519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:42.480017 21519 log.cc:824] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:42.484087 21519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852: No bootstrap required, opened a new log
I20260923 19:34:42.484544 21522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:42.460171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:59111 (local address 127.0.0.1:44461)
0923 19:34:42.461133 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:42.461161 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:42.461226 (+ 65us) server_negotiation.cc:407] Connection header received
0923 19:34:42.463397 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:42.463433 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:42.464110 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:42.464667 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:42.465984 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:42.468074 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:42.469526 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:42.470737 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:42.470947 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:42.474958 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:42.474970 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:42.474992 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:42.475046 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:42.479233 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:42.480245 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:42.480253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:42.480262 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:42.480498 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:42.481130 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:42.481134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:42.481137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:42.481851 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:42.482185 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:42.482699 (+ 514us) server_negotiation.cc:299] Negotiation successful
0923 19:34:42.483364 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":363,"threads_started":1}
I20260923 19:34:42.488834 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:42.493216 21519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a80320b4f4d159cea1ac628f27852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260923 19:34:42.493921 21519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:42.494055 21519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8a80320b4f4d159cea1ac628f27852, State: Initialized, Role: FOLLOWER
I20260923 19:34:42.494904 21519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a80320b4f4d159cea1ac628f27852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260923 19:34:42.495220 21519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:42.495381 21519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:42.495560 21519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:42.498742 21519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a80320b4f4d159cea1ac628f27852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260923 19:34:42.499393 21519 leader_election.cc:302] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c8a80320b4f4d159cea1ac628f27852; no voters:
I20260923 19:34:42.500152 21519 leader_election.cc:288] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:42.500272 21529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:42.503214 21529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [term 1 LEADER]: Becoming Leader. State: Replica: 2c8a80320b4f4d159cea1ac628f27852, State: Running, Role: LEADER
I20260923 19:34:42.503980 21529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a80320b4f4d159cea1ac628f27852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260923 19:34:42.504771 21519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:42.507436 21530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c8a80320b4f4d159cea1ac628f27852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a80320b4f4d159cea1ac628f27852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } } }
I20260923 19:34:42.507927 21530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:42.509840 21522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:42.493639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:53195 (local address 127.0.0.1:44461)
0923 19:34:42.493813 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:42.493820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:42.494454 (+ 634us) server_negotiation.cc:407] Connection header received
0923 19:34:42.494664 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:42.494670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:42.494779 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:42.494910 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:42.496407 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:42.497617 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:42.499123 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:42.499735 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:42.499870 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:42.502257 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:42.502264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:42.502268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:42.502345 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:42.505574 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:42.506506 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:42.506511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:42.506515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:42.506688 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:42.507317 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:42.507321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:42.507324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:42.507769 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:42.508080 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:42.509231 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0923 19:34:42.509435 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 19:34:42.511945 21534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:42.511750 21531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c8a80320b4f4d159cea1ac628f27852. Latest consensus state: current_term: 1 leader_uuid: "2c8a80320b4f4d159cea1ac628f27852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a80320b4f4d159cea1ac628f27852" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } } }
I20260923 19:34:42.512104 21531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:42.519132 21534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:42.536234 21534 catalog_manager.cc:1347] Generated new cluster ID: 4b8078ba2a38434593334f28d6c2386a
I20260923 19:34:42.536365 21534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:42.548604 21534 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:42.550844 21534 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:42.567278 21534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c8a80320b4f4d159cea1ac628f27852: Generated new TSK 0
I20260923 19:34:42.568212 21534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:42.574764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21452
I20260923 19:34:42.575974 21514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:42.680012 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 21452
W20260923 19:34:42.695291 21437 connection.cc:535] client connection to 127.0.0.1:44461 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44461): Connection reset by peer (error 104)
2026-09-23T19:34:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:42 dist-test-slave-kxfb krb5kdc[21551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:42 dist-test-slave-kxfb krb5kdc[21551](info): set up 2 sockets
Sep 23 19:34:42 dist-test-slave-kxfb krb5kdc[21551](info): commencing operation
krb5kdc: starting...
W20260923 19:34:43.718719 388 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:43 dist-test-slave-kxfb krb5kdc[21551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:44.009626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:44.392138 21564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:44.392503 21564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:44.392606 21564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:44.400385 21564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:44.400538 21564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:44.400595 21564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:44.400663 21564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:44.400712 21564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:44.412115 21564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:44.413762 21564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:44.415370 21564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:44.428602 21569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:44.428654 21572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:44.428694 21570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:44.429843 21564 server_base.cc:1034] running on GCE node
I20260923 19:34:44.430869 21564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:44.432447 21564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:44.433732 21564 hybrid_clock.cc:648] HybridClock initialized: now 1790192084433619 us; error 56 us; skew 500 ppm
I20260923 19:34:44.434221 21564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:44 dist-test-slave-kxfb krb5kdc[21551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:44.441785 21564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:44.443574 21564 webserver.cc:460] Webserver started at http://127.0.0.1:35619/ using document root <none> and password file <none>
I20260923 19:34:44.444315 21564 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:44.444422 21564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:44.444706 21564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:44.447921 21564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "de3ac18ef6cd4ecf8feab25606324540"
format_stamp: "Formatted at 2026-09-23 19:34:44 on dist-test-slave-kxfb"
server_key: "c744867623fea4b38902eb0f3901d406"
server_key_iv: "d6d6382a2aff7b148d50230aabf0be0c"
server_key_version: "encryptionkey@0"
I20260923 19:34:44.448676 21564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "de3ac18ef6cd4ecf8feab25606324540"
format_stamp: "Formatted at 2026-09-23 19:34:44 on dist-test-slave-kxfb"
server_key: "c744867623fea4b38902eb0f3901d406"
server_key_iv: "d6d6382a2aff7b148d50230aabf0be0c"
server_key_version: "encryptionkey@0"
I20260923 19:34:44.454638 21564 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 19:34:44.458662 21579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:44.460239 21564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:34:44.460412 21564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "de3ac18ef6cd4ecf8feab25606324540"
format_stamp: "Formatted at 2026-09-23 19:34:44 on dist-test-slave-kxfb"
server_key: "c744867623fea4b38902eb0f3901d406"
server_key_iv: "d6d6382a2aff7b148d50230aabf0be0c"
server_key_version: "encryptionkey@0"
I20260923 19:34:44.460592 21564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:44.480019 21564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:44.484140 21564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:44.484377 21564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:44.503491 21564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37143
I20260923 19:34:44.503564 21630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37143 every 8 connection(s)
I20260923 19:34:44.504884 21564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:44.508517 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 21564
I20260923 19:34:44.508730 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:44.509377 388 external_mini_cluster.cc:1428] Setting key ed6eac5c09d48e99a328c125132bfe2c
I20260923 19:34:44.509742 21631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:44.525645 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:44.511495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:54841 (local address 127.0.0.1:37143)
0923 19:34:44.512301 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:44.512314 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:44.512363 (+ 49us) server_negotiation.cc:407] Connection header received
0923 19:34:44.515076 (+ 2713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:44.515103 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:44.515599 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:44.515985 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:44.516920 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:44.518452 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:44.520611 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:44.521349 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:44.521498 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:44.523081 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:44.523089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:44.523104 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:44.523744 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:44.523837 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:44.523867 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:44.524007 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:44.524340 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 19:34:44.524535 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":351,"threads_started":1}
I20260923 19:34:44.526604 21631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540: Bootstrap starting.
I20260923 19:34:44.529810 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:44.530936 21631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:44.532397 21631 log.cc:824] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:44.535632 21631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540: No bootstrap required, opened a new log
I20260923 19:34:44.543308 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:44.535761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:60225 (local address 127.0.0.1:37143)
0923 19:34:44.535943 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:44.535948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:44.536396 (+ 448us) server_negotiation.cc:407] Connection header received
0923 19:34:44.536588 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:44.536593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:44.536674 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:44.536802 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:44.537504 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:44.538709 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:44.540311 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:44.541000 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:44.541140 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:44.542428 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:44.542434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:44.542439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:44.542462 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:44.542522 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:44.542541 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:44.542661 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:44.542892 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 19:34:44.542955 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 19:34:44.545027 21631 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ac18ef6cd4ecf8feab25606324540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260923 19:34:44.545426 21631 raft_consensus.cc:374] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:44.545517 21631 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3ac18ef6cd4ecf8feab25606324540, State: Initialized, Role: FOLLOWER
I20260923 19:34:44.546103 21631 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ac18ef6cd4ecf8feab25606324540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260923 19:34:44.546345 21631 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:44.546471 21631 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:44.546625 21631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:44.548908 21631 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ac18ef6cd4ecf8feab25606324540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260923 19:34:44.549427 21631 leader_election.cc:302] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3ac18ef6cd4ecf8feab25606324540; no voters:
I20260923 19:34:44.550097 21631 leader_election.cc:288] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:44.550355 21641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:44.551236 21641 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [term 1 LEADER]: Becoming Leader. State: Replica: de3ac18ef6cd4ecf8feab25606324540, State: Running, Role: LEADER
I20260923 19:34:44.551951 21641 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ac18ef6cd4ecf8feab25606324540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260923 19:34:44.552387 21631 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:44.555359 21643 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3ac18ef6cd4ecf8feab25606324540. Latest consensus state: current_term: 1 leader_uuid: "de3ac18ef6cd4ecf8feab25606324540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ac18ef6cd4ecf8feab25606324540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } } }
I20260923 19:34:44.555923 21643 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:44.556283 21642 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de3ac18ef6cd4ecf8feab25606324540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ac18ef6cd4ecf8feab25606324540" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } } }
I20260923 19:34:44.556648 21642 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:44.561995 21648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:44.568575 21648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:44.579911 21648 catalog_manager.cc:1347] Generated new cluster ID: 69b59355d67c491f96e1b529d0fa84ed
I20260923 19:34:44.580035 21648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:44.591764 21648 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:44.593964 21648 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:44.602161 21648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de3ac18ef6cd4ecf8feab25606324540: Generated new TSK 0
I20260923 19:34:44.603509 21648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:44.612851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21564
W20260923 19:34:44.614955 21626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.171:54841
W20260923 19:34:44.615778 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21564: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:44.615952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 21564
W20260923 19:34:44.631301 21549 connection.cc:535] client connection to 127.0.0.1:37143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37143): Connection reset by peer (error 104)
2026-09-23T19:34:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:44 dist-test-slave-kxfb krb5kdc[21663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:44 dist-test-slave-kxfb krb5kdc[21663](info): set up 2 sockets
Sep 23 19:34:44 dist-test-slave-kxfb krb5kdc[21663](info): commencing operation
krb5kdc: starting...
W20260923 19:34:45.666930 388 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:45 dist-test-slave-kxfb krb5kdc[21663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:45.965994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:46.350071 21676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:46.350514 21676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:46.350675 21676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:46.358711 21676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:46.358860 21676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:46.358942 21676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:46.359033 21676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:46.359097 21676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:46.370229 21676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:46.371955 21676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:46.373805 21676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:46.387465 21681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:46.387465 21682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:46.387547 21684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:46.388787 21676 server_base.cc:1034] running on GCE node
I20260923 19:34:46.389707 21676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:46.391542 21676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:46.392771 21676 hybrid_clock.cc:648] HybridClock initialized: now 1790192086392715 us; error 51 us; skew 500 ppm
I20260923 19:34:46.393177 21676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:46 dist-test-slave-kxfb krb5kdc[21663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:46.397043 21676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:46.399017 21676 webserver.cc:460] Webserver started at http://127.0.0.1:42375/ using document root <none> and password file <none>
I20260923 19:34:46.399847 21676 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:46.399976 21676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:46.400317 21676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:46.404016 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "c41dfb3635964de58d791d9c1f9ff644"
format_stamp: "Formatted at 2026-09-23 19:34:46 on dist-test-slave-kxfb"
server_key: "62497d6f6dc06890c1d21c98fd2adfda"
server_key_iv: "a7d2bd1ca2530c37e4d8d37758c2df03"
server_key_version: "encryptionkey@0"
I20260923 19:34:46.404896 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "c41dfb3635964de58d791d9c1f9ff644"
format_stamp: "Formatted at 2026-09-23 19:34:46 on dist-test-slave-kxfb"
server_key: "62497d6f6dc06890c1d21c98fd2adfda"
server_key_iv: "a7d2bd1ca2530c37e4d8d37758c2df03"
server_key_version: "encryptionkey@0"
I20260923 19:34:46.411240 21676 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 19:34:46.415428 21691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:46.417066 21676 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:34:46.417263 21676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "c41dfb3635964de58d791d9c1f9ff644"
format_stamp: "Formatted at 2026-09-23 19:34:46 on dist-test-slave-kxfb"
server_key: "62497d6f6dc06890c1d21c98fd2adfda"
server_key_iv: "a7d2bd1ca2530c37e4d8d37758c2df03"
server_key_version: "encryptionkey@0"
I20260923 19:34:46.417503 21676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:46.424173 21676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:46.428653 21676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:46.428958 21676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:46.449270 21676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36887
I20260923 19:34:46.449391 21742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36887 every 8 connection(s)
I20260923 19:34:46.451004 21676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:46.454838 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 21676
I20260923 19:34:46.455060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:46.455333 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:46.455569 388 external_mini_cluster.cc:1428] Setting key 4863574547ea42baebf836b2d700f5f0
I20260923 19:34:46.464438 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:46.457551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:56025 (local address 127.0.0.1:36887)
0923 19:34:46.458509 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:46.458527 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:46.458580 (+ 53us) server_negotiation.cc:407] Connection header received
0923 19:34:46.460444 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:46.460472 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:46.460922 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:46.461335 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:46.462615 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:46.462622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:46.462637 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:46.462653 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:46.462729 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:46.462758 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:46.462963 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:46.463090 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 19:34:46.463332 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":400,"threads_started":1}
I20260923 19:34:46.469208 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:46.470211 21743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644: Bootstrap starting.
I20260923 19:34:46.473552 21743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:46.474962 21743 log.cc:824] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:46.476884 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:46.474320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:49425 (local address 127.0.0.1:36887)
0923 19:34:46.474526 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:46.474533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:46.474917 (+ 384us) server_negotiation.cc:407] Connection header received
0923 19:34:46.475120 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:46.475126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:46.475201 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:46.475290 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:46.476227 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:46.476233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:46.476238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:46.476257 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:46.476300 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:46.476316 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:46.476428 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:46.476516 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 19:34:46.476588 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 19:34:46.478040 21743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644: No bootstrap required, opened a new log
I20260923 19:34:46.485852 21743 raft_consensus.cc:348] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41dfb3635964de58d791d9c1f9ff644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36887 } }
I20260923 19:34:46.486306 21743 raft_consensus.cc:374] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:46.486408 21743 raft_consensus.cc:729] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41dfb3635964de58d791d9c1f9ff644, State: Initialized, Role: FOLLOWER
I20260923 19:34:46.487005 21743 consensus_queue.cc:260] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41dfb3635964de58d791d9c1f9ff644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36887 } }
I20260923 19:34:46.487236 21743 raft_consensus.cc:388] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:46.487368 21743 raft_consensus.cc:482] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:46.487516 21743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:46.489735 21743 raft_consensus.cc:504] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41dfb3635964de58d791d9c1f9ff644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36887 } }
I20260923 19:34:46.490279 21743 leader_election.cc:302] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c41dfb3635964de58d791d9c1f9ff644; no voters:
I20260923 19:34:46.491034 21743 leader_election.cc:288] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:46.491273 21753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:46.492436 21753 raft_consensus.cc:686] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [term 1 LEADER]: Becoming Leader. State: Replica: c41dfb3635964de58d791d9c1f9ff644, State: Running, Role: LEADER
I20260923 19:34:46.493155 21753 consensus_queue.cc:237] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41dfb3635964de58d791d9c1f9ff644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36887 } }
I20260923 19:34:46.493345 21743 sys_catalog.cc:564] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:46.497035 21754 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c41dfb3635964de58d791d9c1f9ff644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41dfb3635964de58d791d9c1f9ff644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36887 } } }
I20260923 19:34:46.496999 21755 sys_catalog.cc:455] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c41dfb3635964de58d791d9c1f9ff644. Latest consensus state: current_term: 1 leader_uuid: "c41dfb3635964de58d791d9c1f9ff644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c41dfb3635964de58d791d9c1f9ff644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36887 } } }
I20260923 19:34:46.497557 21754 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:46.497561 21755 sys_catalog.cc:458] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:46.502074 21761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:46.507467 21761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:46.518561 21761 catalog_manager.cc:1347] Generated new cluster ID: a99dd8c876eb4691b63a96f804f09537
I20260923 19:34:46.518719 21761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:46.544873 21761 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:46.545644 21761 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:46.561511 21761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c41dfb3635964de58d791d9c1f9ff644: Generated new TSK 0
I20260923 19:34:46.562825 21761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:46.567788 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21676
W20260923 19:34:46.569386 21738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.171:56025
W20260923 19:34:46.570269 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:46.570482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 21676
2026-09-23T19:34:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:46 dist-test-slave-kxfb krb5kdc[21774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:46 dist-test-slave-kxfb krb5kdc[21774](info): set up 2 sockets
Sep 23 19:34:46 dist-test-slave-kxfb krb5kdc[21774](info): commencing operation
krb5kdc: starting...
W20260923 19:34:47.618822 388 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:47 dist-test-slave-kxfb krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:47.910717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:48.290261 21787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:48.290709 21787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:48.290827 21787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:48.298771 21787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:48.298918 21787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:48.298969 21787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:48.299041 21787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:48.299090 21787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:48.310002 21787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:48.311647 21787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:48.313107 21787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:48.326180 21795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:48.326261 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:48.326241 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:48.327257 21787 server_base.cc:1034] running on GCE node
I20260923 19:34:48.328207 21787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:48.329846 21787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:48.331067 21787 hybrid_clock.cc:648] HybridClock initialized: now 1790192088331013 us; error 44 us; skew 500 ppm
I20260923 19:34:48.331449 21787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:48 dist-test-slave-kxfb krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:48.334787 21787 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:48.336411 21787 webserver.cc:460] Webserver started at http://127.0.0.1:33197/ using document root <none> and password file <none>
I20260923 19:34:48.337155 21787 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:48.337275 21787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:48.337594 21787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:48.340754 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "ca922685be724c2fba51eb8496380b82"
format_stamp: "Formatted at 2026-09-23 19:34:48 on dist-test-slave-kxfb"
server_key: "91ed62d94d23cc63efd3057f04d2d844"
server_key_iv: "6e87aec41e2a3d9b507175a92480c4d8"
server_key_version: "encryptionkey@0"
I20260923 19:34:48.341468 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "ca922685be724c2fba51eb8496380b82"
format_stamp: "Formatted at 2026-09-23 19:34:48 on dist-test-slave-kxfb"
server_key: "91ed62d94d23cc63efd3057f04d2d844"
server_key_iv: "6e87aec41e2a3d9b507175a92480c4d8"
server_key_version: "encryptionkey@0"
I20260923 19:34:48.347265 21787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 19:34:48.351905 21802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:48.353390 21787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260923 19:34:48.353570 21787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "ca922685be724c2fba51eb8496380b82"
format_stamp: "Formatted at 2026-09-23 19:34:48 on dist-test-slave-kxfb"
server_key: "91ed62d94d23cc63efd3057f04d2d844"
server_key_iv: "6e87aec41e2a3d9b507175a92480c4d8"
server_key_version: "encryptionkey@0"
I20260923 19:34:48.353761 21787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:48.370535 21787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:48.374781 21787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:48.375054 21787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:48.393857 21787 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43669
I20260923 19:34:48.393908 21853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43669 every 8 connection(s)
I20260923 19:34:48.395385 21787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:48.399008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 21787
I20260923 19:34:48.399187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:48.399603 388 external_mini_cluster.cc:1428] Setting key bbc748f36709e649c5f92f552ef8f26e
I20260923 19:34:48.400257 21854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 19:34:48 dist-test-slave-kxfb krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 19:34:48.416242 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82: Bootstrap starting.
I20260923 19:34:48.420924 21854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:48.422245 21854 log.cc:824] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:48.426026 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:48.401689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:48743 (local address 127.0.0.1:43669)
0923 19:34:48.402603 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:48.402622 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:48.402692 (+ 70us) server_negotiation.cc:407] Connection header received
0923 19:34:48.404678 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:48.404711 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:48.405458 (+ 747us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:48.406020 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:48.407396 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:48.409564 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:48.411026 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:48.412080 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:48.412281 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:48.416614 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:48.416627 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:48.416651 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:48.416702 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:48.420869 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:48.421722 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:48.421731 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:48.421740 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:48.421964 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:48.422468 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:48.422472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:48.422475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:48.423271 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:48.423563 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:48.424145 (+ 582us) server_negotiation.cc:299] Negotiation successful
0923 19:34:48.424543 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":337,"threads_started":1}
I20260923 19:34:48.426366 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82: No bootstrap required, opened a new log
I20260923 19:34:48.430599 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:48.437036 21854 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca922685be724c2fba51eb8496380b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43669 } }
I20260923 19:34:48.437582 21854 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:48.437702 21854 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca922685be724c2fba51eb8496380b82, State: Initialized, Role: FOLLOWER
I20260923 19:34:48.438573 21854 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca922685be724c2fba51eb8496380b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43669 } }
I20260923 19:34:48.438885 21854 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:48.439051 21854 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:48.439253 21854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:48.442577 21854 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca922685be724c2fba51eb8496380b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43669 } }
I20260923 19:34:48.443210 21854 leader_election.cc:302] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca922685be724c2fba51eb8496380b82; no voters:
I20260923 19:34:48.443975 21854 leader_election.cc:288] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:48.444178 21864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:48.444864 21864 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [term 1 LEADER]: Becoming Leader. State: Replica: ca922685be724c2fba51eb8496380b82, State: Running, Role: LEADER
I20260923 19:34:48.445559 21864 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca922685be724c2fba51eb8496380b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43669 } }
I20260923 19:34:48.447813 21865 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca922685be724c2fba51eb8496380b82. Latest consensus state: current_term: 1 leader_uuid: "ca922685be724c2fba51eb8496380b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca922685be724c2fba51eb8496380b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43669 } } }
I20260923 19:34:48.448241 21865 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:48.448637 21866 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca922685be724c2fba51eb8496380b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca922685be724c2fba51eb8496380b82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43669 } } }
I20260923 19:34:48.448982 21866 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:48.449244 21854 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:48.455250 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:48.436554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:60637 (local address 127.0.0.1:43669)
0923 19:34:48.436822 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:48.436828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:48.437199 (+ 371us) server_negotiation.cc:407] Connection header received
0923 19:34:48.437398 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:48.437402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:48.437475 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:48.437633 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:48.439701 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:48.440617 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:48.442473 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:48.443047 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:48.443153 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:48.446700 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:48.446707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:48.446712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:48.446759 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:48.451024 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:48.452259 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:48.452265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:48.452269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:48.452481 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:48.453181 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:48.453186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:48.453190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:48.453616 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:48.454036 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:48.454571 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 19:34:48.454795 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260923 19:34:48.459679 21868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:48.464960 21868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:48.475472 21868 catalog_manager.cc:1347] Generated new cluster ID: 306a826b3d3147ad9e8748e59b6830a6
I20260923 19:34:48.475626 21868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:48.520901 21868 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:48.522235 21868 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:48.539609 21868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca922685be724c2fba51eb8496380b82: Generated new TSK 0
I20260923 19:34:48.540462 21868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:48.548028 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21787
I20260923 19:34:48.549240 21849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:48.645274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 21787
W20260923 19:34:48.663232 21772 connection.cc:535] client connection to 127.0.0.1:43669 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43669): Connection reset by peer (error 104)
2026-09-23T19:34:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2076 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:48 dist-test-slave-kxfb krb5kdc[21886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:48 dist-test-slave-kxfb krb5kdc[21886](info): set up 2 sockets
Sep 23 19:34:48 dist-test-slave-kxfb krb5kdc[21886](info): commencing operation
krb5kdc: starting...
W20260923 19:34:49.687237 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:49 dist-test-slave-kxfb krb5kdc[21886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:49.977119 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:50.362411 21899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:50.363041 21899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:50.363215 21899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:50.372054 21899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:50.372195 21899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:50.372253 21899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:50.372325 21899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:50.372376 21899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:50.383503 21899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:50.385251 21899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:50.386857 21899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:50.400297 21904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:50.400338 21905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:50.400530 21907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:50.401080 21899 server_base.cc:1034] running on GCE node
I20260923 19:34:50.402117 21899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:50.403726 21899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:50.404963 21899 hybrid_clock.cc:648] HybridClock initialized: now 1790192090404913 us; error 43 us; skew 500 ppm
I20260923 19:34:50.405362 21899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:50 dist-test-slave-kxfb krb5kdc[21886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:50.413698 21899 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:50.415495 21899 webserver.cc:460] Webserver started at http://127.0.0.1:36289/ using document root <none> and password file <none>
I20260923 19:34:50.416257 21899 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:50.416378 21899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:50.416707 21899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:50.419955 21899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "f3d51cf164fe4e36a8578fe984a1046b"
format_stamp: "Formatted at 2026-09-23 19:34:50 on dist-test-slave-kxfb"
server_key: "29f976e0197e98fec03e0776d3ae26f6"
server_key_iv: "ef5fb98201e4f500a5af7681d098197f"
server_key_version: "encryptionkey@0"
I20260923 19:34:50.420735 21899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "f3d51cf164fe4e36a8578fe984a1046b"
format_stamp: "Formatted at 2026-09-23 19:34:50 on dist-test-slave-kxfb"
server_key: "29f976e0197e98fec03e0776d3ae26f6"
server_key_iv: "ef5fb98201e4f500a5af7681d098197f"
server_key_version: "encryptionkey@0"
I20260923 19:34:50.426725 21899 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260923 19:34:50.430656 21914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:50.432215 21899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:34:50.432479 21899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "f3d51cf164fe4e36a8578fe984a1046b"
format_stamp: "Formatted at 2026-09-23 19:34:50 on dist-test-slave-kxfb"
server_key: "29f976e0197e98fec03e0776d3ae26f6"
server_key_iv: "ef5fb98201e4f500a5af7681d098197f"
server_key_version: "encryptionkey@0"
I20260923 19:34:50.432691 21899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:50.467896 21899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:50.472427 21899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:50.472702 21899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:50.492165 21899 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40697
I20260923 19:34:50.492218 21965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40697 every 8 connection(s)
I20260923 19:34:50.493589 21899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:50.498227 21966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:50.499183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 21899
I20260923 19:34:50.499403 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:50.500046 388 external_mini_cluster.cc:1428] Setting key 03d35cca3354b2d4ea142d5cf9840cdc
I20260923 19:34:50.514097 21966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b: Bootstrap starting.
I20260923 19:34:50.515436 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:50.502435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:51265 (local address 127.0.0.1:40697)
0923 19:34:50.503327 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:50.503340 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:50.503394 (+ 54us) server_negotiation.cc:407] Connection header received
0923 19:34:50.505366 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:50.505391 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:50.505879 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:50.506261 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:50.507019 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:50.508283 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:50.510366 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:50.511217 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:50.511391 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:50.512896 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:50.512905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:50.512920 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:50.513556 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:50.513654 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:50.513680 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:50.513812 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:50.514098 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 19:34:50.514324 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":363,"threads_started":1}
I20260923 19:34:50.518083 21966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:50.519456 21966 log.cc:824] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:50.519567 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:50.522760 21966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b: No bootstrap required, opened a new log
I20260923 19:34:50.531342 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:50.523978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:45289 (local address 127.0.0.1:40697)
0923 19:34:50.524165 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:50.524170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:50.524552 (+ 382us) server_negotiation.cc:407] Connection header received
0923 19:34:50.524791 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:50.524795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:50.524868 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:50.524950 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:50.525622 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:50.526525 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:50.528475 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:50.528967 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:50.529087 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:50.530450 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:50.530455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:50.530458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:50.530480 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:50.530539 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:50.530555 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:50.530657 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:50.530921 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 19:34:50.530999 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 19:34:50.533025 21966 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d51cf164fe4e36a8578fe984a1046b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } }
I20260923 19:34:50.533490 21966 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:50.533583 21966 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d51cf164fe4e36a8578fe984a1046b, State: Initialized, Role: FOLLOWER
I20260923 19:34:50.534251 21966 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d51cf164fe4e36a8578fe984a1046b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } }
I20260923 19:34:50.534540 21966 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:50.534678 21966 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:50.534844 21966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:50.537676 21966 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d51cf164fe4e36a8578fe984a1046b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } }
I20260923 19:34:50.538240 21966 leader_election.cc:302] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3d51cf164fe4e36a8578fe984a1046b; no voters:
I20260923 19:34:50.538954 21966 leader_election.cc:288] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:50.539141 21976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:50.540062 21976 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [term 1 LEADER]: Becoming Leader. State: Replica: f3d51cf164fe4e36a8578fe984a1046b, State: Running, Role: LEADER
I20260923 19:34:50.540724 21976 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d51cf164fe4e36a8578fe984a1046b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } }
I20260923 19:34:50.541456 21966 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:50.543745 21977 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3d51cf164fe4e36a8578fe984a1046b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d51cf164fe4e36a8578fe984a1046b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } } }
I20260923 19:34:50.544735 21977 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:50.544400 21978 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3d51cf164fe4e36a8578fe984a1046b. Latest consensus state: current_term: 1 leader_uuid: "f3d51cf164fe4e36a8578fe984a1046b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d51cf164fe4e36a8578fe984a1046b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } } }
I20260923 19:34:50.545356 21978 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:50.551129 21983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:50.556339 21983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:50.566565 21983 catalog_manager.cc:1347] Generated new cluster ID: 1706325b21ff4bb9825c01ff39dd7390
I20260923 19:34:50.566694 21983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:50.577561 21983 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:50.578962 21983 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:50.590150 21983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3d51cf164fe4e36a8578fe984a1046b: Generated new TSK 0
I20260923 19:34:50.591032 21983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:50.598325 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21899
W20260923 19:34:50.599680 21961 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.171:51265
W20260923 19:34:50.600409 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 21899: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:50.600539 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 21899
W20260923 19:34:50.617682 21884 connection.cc:535] client connection to 127.0.0.1:40697 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40697): Connection reset by peer (error 104)
2026-09-23T19:34:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:50 dist-test-slave-kxfb krb5kdc[21997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:50 dist-test-slave-kxfb krb5kdc[21997](info): set up 2 sockets
Sep 23 19:34:50 dist-test-slave-kxfb krb5kdc[21997](info): commencing operation
krb5kdc: starting...
W20260923 19:34:51.639053 388 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:51 dist-test-slave-kxfb krb5kdc[21997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 19:34:51.933305 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:52.317225 22010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:52.317605 22010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:52.317704 22010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:52.325501 22010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:52.325644 22010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:52.325693 22010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:52.325766 22010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:52.325824 22010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:52.336633 22010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:52.338358 22010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:52.339876 22010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:52.352880 22018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:52.352926 22016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:52.353008 22015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:52.354202 22010 server_base.cc:1034] running on GCE node
I20260923 19:34:52.355140 22010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:52.356722 22010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:52.357985 22010 hybrid_clock.cc:648] HybridClock initialized: now 1790192092357921 us; error 51 us; skew 500 ppm
I20260923 19:34:52.358424 22010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:52 dist-test-slave-kxfb krb5kdc[21997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:52.361676 22010 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 19:34:52.363324 22010 webserver.cc:460] Webserver started at http://127.0.0.1:37633/ using document root <none> and password file <none>
I20260923 19:34:52.364046 22010 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:52.364156 22010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:52.364461 22010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:52.367527 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "890be6691dc9411b9674ad3217d0e7e9"
format_stamp: "Formatted at 2026-09-23 19:34:52 on dist-test-slave-kxfb"
server_key: "8c694dafa807f9e401f0915a8354ac94"
server_key_iv: "a8780907bd98dc617f8c4ceacf9e54c7"
server_key_version: "encryptionkey@0"
I20260923 19:34:52.368258 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "890be6691dc9411b9674ad3217d0e7e9"
format_stamp: "Formatted at 2026-09-23 19:34:52 on dist-test-slave-kxfb"
server_key: "8c694dafa807f9e401f0915a8354ac94"
server_key_iv: "a8780907bd98dc617f8c4ceacf9e54c7"
server_key_version: "encryptionkey@0"
I20260923 19:34:52.375206 22010 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 19:34:52.378937 22025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:52.380661 22010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:34:52.380863 22010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "890be6691dc9411b9674ad3217d0e7e9"
format_stamp: "Formatted at 2026-09-23 19:34:52 on dist-test-slave-kxfb"
server_key: "8c694dafa807f9e401f0915a8354ac94"
server_key_iv: "a8780907bd98dc617f8c4ceacf9e54c7"
server_key_version: "encryptionkey@0"
I20260923 19:34:52.381095 22010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:52.387274 22010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:52.391546 22010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:52.391834 22010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:52.411222 22010 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40777
I20260923 19:34:52.411331 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40777 every 8 connection(s)
I20260923 19:34:52.412736 22010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:52.413493 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22010
I20260923 19:34:52.413697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:52.414242 388 external_mini_cluster.cc:1428] Setting key a6436785822dd3ce2bdabb70a97e86be
I20260923 19:34:52.418469 22077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:52.424175 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:52.416306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:38871 (local address 127.0.0.1:40777)
0923 19:34:52.417229 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:52.417260 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:52.417324 (+ 64us) server_negotiation.cc:407] Connection header received
0923 19:34:52.419879 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:52.419914 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:52.420402 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:52.420935 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:52.422001 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:52.422010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:52.422032 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:52.422053 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:52.422149 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:52.422180 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:52.422455 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:52.422602 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 19:34:52.422825 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":347,"threads_started":1}
I20260923 19:34:52.428406 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:52.432091 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9: Bootstrap starting.
I20260923 19:34:52.435184 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:52.432993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:43311 (local address 127.0.0.1:40777)
0923 19:34:52.433195 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:52.433201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:52.433373 (+ 172us) server_negotiation.cc:407] Connection header received
0923 19:34:52.433514 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:52.433520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:52.433595 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:52.433738 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:52.434567 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:52.434573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:52.434578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:52.434596 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:52.434643 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:52.434662 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:52.434782 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:52.434830 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 19:34:52.434892 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 19:34:52.436105 22077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:52.437376 22077 log.cc:824] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:52.440765 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9: No bootstrap required, opened a new log
I20260923 19:34:52.449589 22077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890be6691dc9411b9674ad3217d0e7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } }
I20260923 19:34:52.450071 22077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:52.450201 22077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890be6691dc9411b9674ad3217d0e7e9, State: Initialized, Role: FOLLOWER
I20260923 19:34:52.450970 22077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890be6691dc9411b9674ad3217d0e7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } }
I20260923 19:34:52.451251 22077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:52.451417 22077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:52.451607 22077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:52.454452 22077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890be6691dc9411b9674ad3217d0e7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } }
I20260923 19:34:52.455120 22077 leader_election.cc:302] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890be6691dc9411b9674ad3217d0e7e9; no voters:
I20260923 19:34:52.455824 22077 leader_election.cc:288] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:52.455989 22087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:52.456835 22087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [term 1 LEADER]: Becoming Leader. State: Replica: 890be6691dc9411b9674ad3217d0e7e9, State: Running, Role: LEADER
I20260923 19:34:52.457525 22087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890be6691dc9411b9674ad3217d0e7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } }
I20260923 19:34:52.458324 22077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:52.460547 22088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890be6691dc9411b9674ad3217d0e7e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890be6691dc9411b9674ad3217d0e7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } } }
I20260923 19:34:52.462332 22088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:52.460912 22089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890be6691dc9411b9674ad3217d0e7e9. Latest consensus state: current_term: 1 leader_uuid: "890be6691dc9411b9674ad3217d0e7e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890be6691dc9411b9674ad3217d0e7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40777 } } }
I20260923 19:34:52.462946 22089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:52.464071 22093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:52.470100 22093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:52.481137 22093 catalog_manager.cc:1347] Generated new cluster ID: 5483901c940c4bc4a89dc0b62b0e4b73
I20260923 19:34:52.481264 22093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:52.501334 22093 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:52.501917 22093 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:52.514717 22093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890be6691dc9411b9674ad3217d0e7e9: Generated new TSK 0
I20260923 19:34:52.515970 22093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:52.521457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22010
W20260923 19:34:52.523487 22072 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.171:38871
W20260923 19:34:52.524271 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22010: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:52.524446 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22010
2026-09-23T19:34:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:52 dist-test-slave-kxfb krb5kdc[22108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:52 dist-test-slave-kxfb krb5kdc[22108](info): set up 2 sockets
Sep 23 19:34:52 dist-test-slave-kxfb krb5kdc[22108](info): commencing operation
krb5kdc: starting...
W20260923 19:34:53.562932 388 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:53 dist-test-slave-kxfb krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:34:53.845703 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33285
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:54.189083 22121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:54.189399 22121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:54.189524 22121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:54.196811 22121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:54.196944 22121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:54.196993 22121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:54.197078 22121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:54.197135 22121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:54.207372 22121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:54.208765 22121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:54.210130 22121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:54.224290 22126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:54.224313 22127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:54.224114 22129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:54.225243 22121 server_base.cc:1034] running on GCE node
I20260923 19:34:54.226130 22121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:54.227679 22121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:54.228911 22121 hybrid_clock.cc:648] HybridClock initialized: now 1790192094228851 us; error 55 us; skew 500 ppm
I20260923 19:34:54.229275 22121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:54 dist-test-slave-kxfb krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:54.232623 22121 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:34:54.234246 22121 webserver.cc:460] Webserver started at http://127.0.97.62:39403/ using document root <none> and password file <none>
I20260923 19:34:54.234958 22121 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:54.235073 22121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:54.235353 22121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:54.238260 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5"
format_stamp: "Formatted at 2026-09-23 19:34:54 on dist-test-slave-kxfb"
server_key: "21daca3233d3f262af6587f05f031313"
server_key_iv: "9bb42084213c30b81a0fffe81fb8f0b0"
server_key_version: "encryptionkey@0"
I20260923 19:34:54.239040 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5"
format_stamp: "Formatted at 2026-09-23 19:34:54 on dist-test-slave-kxfb"
server_key: "21daca3233d3f262af6587f05f031313"
server_key_iv: "9bb42084213c30b81a0fffe81fb8f0b0"
server_key_version: "encryptionkey@0"
I20260923 19:34:54.244526 22121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:34:54.248239 22136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:54.249715 22121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 19:34:54.249907 22121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5"
format_stamp: "Formatted at 2026-09-23 19:34:54 on dist-test-slave-kxfb"
server_key: "21daca3233d3f262af6587f05f031313"
server_key_iv: "9bb42084213c30b81a0fffe81fb8f0b0"
server_key_version: "encryptionkey@0"
I20260923 19:34:54.250088 22121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:54.277868 22121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:54.282203 22121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:54.282521 22121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:54.301506 22121 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33285
I20260923 19:34:54.301577 22187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33285 every 8 connection(s)
I20260923 19:34:54.302929 22121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:54.304411 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22121
I20260923 19:34:54.304572 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:54.304982 388 external_mini_cluster.cc:1428] Setting key 0bf0e01819f9d848854fadda75293939
I20260923 19:34:54.308058 22188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 19:34:54 dist-test-slave-kxfb krb5kdc[22108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:34:54.322139 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5: Bootstrap starting.
I20260923 19:34:54.326558 22188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:54.327739 22188 log.cc:824] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:54.331022 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:54.306974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48581 (local address 127.0.97.62:33285)
0923 19:34:54.307864 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:54.307882 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:54.307943 (+ 61us) server_negotiation.cc:407] Connection header received
0923 19:34:54.310362 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:54.310392 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:54.310964 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:54.311488 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:54.312751 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:54.314887 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:54.316209 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:54.317253 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:54.317445 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:54.321816 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:54.321829 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:54.321850 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:54.321906 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:54.325924 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:54.326952 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:54.326963 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:54.326975 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:54.327233 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:54.327702 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:54.327706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:54.327708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:54.328560 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:54.328931 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:54.329486 (+ 555us) server_negotiation.cc:299] Negotiation successful
0923 19:34:54.329866 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":342,"threads_started":1}
I20260923 19:34:54.331301 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5: No bootstrap required, opened a new log
I20260923 19:34:54.335031 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:54.341496 22188 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260923 19:34:54.342093 22188 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:54.342204 22188 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a3e3f197cd4b55b2a4e6a4339d91f5, State: Initialized, Role: FOLLOWER
I20260923 19:34:54.342954 22188 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260923 19:34:54.343185 22188 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:54.343343 22188 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:54.343519 22188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:54.345997 22188 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260923 19:34:54.346632 22188 leader_election.cc:302] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1a3e3f197cd4b55b2a4e6a4339d91f5; no voters:
I20260923 19:34:54.347294 22188 leader_election.cc:288] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:54.347472 22198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:54.349890 22188 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:54.350381 22198 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [term 1 LEADER]: Becoming Leader. State: Replica: e1a3e3f197cd4b55b2a4e6a4339d91f5, State: Running, Role: LEADER
I20260923 19:34:54.351099 22198 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260923 19:34:54.359799 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } } }
I20260923 19:34:54.360229 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:54.360402 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:54.340907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39801 (local address 127.0.97.62:33285)
0923 19:34:54.341129 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:54.341135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:54.341421 (+ 286us) server_negotiation.cc:407] Connection header received
0923 19:34:54.341645 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:54.341651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:54.341756 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:54.341900 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:54.343604 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:54.344502 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:54.345746 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:54.346248 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:54.346400 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:54.349727 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:54.349735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:54.349740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:34:54.349791 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:54.354450 (+ 4659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:54.355712 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:54.355730 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:54.355734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:54.355912 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:34:54.357170 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:34:54.357176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:34:54.357181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:34:54.357733 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:54.358080 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:54.359570 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0923 19:34:54.359885 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 19:34:54.360764 22200 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1a3e3f197cd4b55b2a4e6a4339d91f5. Latest consensus state: current_term: 1 leader_uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a3e3f197cd4b55b2a4e6a4339d91f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } } }
I20260923 19:34:54.361065 22200 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5 [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:54.369513 22206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:54.374703 22206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:54.385250 22206 catalog_manager.cc:1347] Generated new cluster ID: 624f540b846641e3ac46abde4826251e
I20260923 19:34:54.385404 22206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:54.425480 22206 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:54.427573 22206 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:54.439463 22206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1a3e3f197cd4b55b2a4e6a4339d91f5: Generated new TSK 0
I20260923 19:34:54.440424 22206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:54.452126 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22121
I20260923 19:34:54.453650 22183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:34:54.556655 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22121
W20260923 19:34:54.572232 22106 connection.cc:535] client connection to 127.0.97.62:33285 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:33285): Connection reset by peer (error 104)
2026-09-23T19:34:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:54 dist-test-slave-kxfb krb5kdc[22220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:54 dist-test-slave-kxfb krb5kdc[22220](info): set up 2 sockets
Sep 23 19:34:54 dist-test-slave-kxfb krb5kdc[22220](info): commencing operation
krb5kdc: starting...
W20260923 19:34:55.594671 388 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:55 dist-test-slave-kxfb krb5kdc[22220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:34:55.884044 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41053
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:56.205506 22233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:56.205804 22233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:56.205927 22233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:56.213804 22233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:56.213943 22233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:56.213997 22233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:56.214084 22233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:56.214150 22233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:56.224599 22233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:56.225955 22233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:56.227285 22233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:56.240180 22238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:56.240362 22239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:56.240396 22241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:56.241668 22233 server_base.cc:1034] running on GCE node
I20260923 19:34:56.242712 22233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:56.244457 22233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:56.245692 22233 hybrid_clock.cc:648] HybridClock initialized: now 1790192096245624 us; error 54 us; skew 500 ppm
I20260923 19:34:56.246086 22233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:56 dist-test-slave-kxfb krb5kdc[22220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:56.254585 22233 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:34:56.256579 22233 webserver.cc:460] Webserver started at http://127.0.97.62:44071/ using document root <none> and password file <none>
I20260923 19:34:56.257458 22233 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:56.257576 22233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:56.257918 22233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:56.261458 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "6fae094e2a2540ab8b498f1985ac188d"
format_stamp: "Formatted at 2026-09-23 19:34:56 on dist-test-slave-kxfb"
server_key: "e1a1a21023302107c4175a2a55d17ea4"
server_key_iv: "8155155456df7d1bd70dc280bbc51ee2"
server_key_version: "encryptionkey@0"
I20260923 19:34:56.262272 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "6fae094e2a2540ab8b498f1985ac188d"
format_stamp: "Formatted at 2026-09-23 19:34:56 on dist-test-slave-kxfb"
server_key: "e1a1a21023302107c4175a2a55d17ea4"
server_key_iv: "8155155456df7d1bd70dc280bbc51ee2"
server_key_version: "encryptionkey@0"
I20260923 19:34:56.268518 22233 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 19:34:56.272867 22248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:56.274701 22233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 19:34:56.274920 22233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "6fae094e2a2540ab8b498f1985ac188d"
format_stamp: "Formatted at 2026-09-23 19:34:56 on dist-test-slave-kxfb"
server_key: "e1a1a21023302107c4175a2a55d17ea4"
server_key_iv: "8155155456df7d1bd70dc280bbc51ee2"
server_key_version: "encryptionkey@0"
I20260923 19:34:56.275146 22233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:56.320659 22233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:56.325604 22233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:56.325924 22233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:56.346159 22233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41053
I20260923 19:34:56.346186 22299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41053 every 8 connection(s)
I20260923 19:34:56.347826 22233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:56.351840 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:56.354214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22233
I20260923 19:34:56.354427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:56.354926 388 external_mini_cluster.cc:1428] Setting key cb8b883a091a0b2dee3d70007ffb548e
I20260923 19:34:56.365187 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d: Bootstrap starting.
I20260923 19:34:56.368966 22300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:56.370328 22300 log.cc:824] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:56.371503 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:56.357243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34609 (local address 127.0.97.62:41053)
0923 19:34:56.358198 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:56.358216 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:56.358278 (+ 62us) server_negotiation.cc:407] Connection header received
0923 19:34:56.360613 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:56.360644 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:56.361227 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:56.361710 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:56.362371 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:56.364337 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:56.366006 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:56.367077 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:56.367318 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:56.368617 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:56.368629 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:56.368649 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:56.369426 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:56.369548 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:56.369582 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:56.369747 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:56.370036 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 19:34:56.370229 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":396,"threads_started":1}
I20260923 19:34:56.373816 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d: No bootstrap required, opened a new log
I20260923 19:34:56.376610 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:56.382684 22300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fae094e2a2540ab8b498f1985ac188d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41053 } }
I20260923 19:34:56.383163 22300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:56.383266 22300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fae094e2a2540ab8b498f1985ac188d, State: Initialized, Role: FOLLOWER
I20260923 19:34:56.383934 22300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fae094e2a2540ab8b498f1985ac188d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41053 } }
I20260923 19:34:56.384192 22300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:56.384330 22300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:56.384510 22300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:56.387669 22300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fae094e2a2540ab8b498f1985ac188d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41053 } }
I20260923 19:34:56.388319 22300 leader_election.cc:302] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fae094e2a2540ab8b498f1985ac188d; no voters:
I20260923 19:34:56.389169 22300 leader_election.cc:288] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:56.390252 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:56.381570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43809 (local address 127.0.97.62:41053)
0923 19:34:56.381770 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:56.381778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:56.382250 (+ 472us) server_negotiation.cc:407] Connection header received
0923 19:34:56.382513 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:56.382518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:56.382595 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:56.382689 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:56.383305 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:56.384243 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:56.386350 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:34:56.386905 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:34:56.387037 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:34:56.389134 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:56.389141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:56.389146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:56.389182 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:56.389272 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:56.389298 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:56.389393 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:56.389771 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 19:34:56.389860 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 19:34:56.390178 22311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:56.391428 22311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [term 1 LEADER]: Becoming Leader. State: Replica: 6fae094e2a2540ab8b498f1985ac188d, State: Running, Role: LEADER
I20260923 19:34:56.391691 22300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:56.392184 22311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fae094e2a2540ab8b498f1985ac188d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41053 } }
I20260923 19:34:56.395641 22312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fae094e2a2540ab8b498f1985ac188d. Latest consensus state: current_term: 1 leader_uuid: "6fae094e2a2540ab8b498f1985ac188d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fae094e2a2540ab8b498f1985ac188d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41053 } } }
I20260923 19:34:56.395792 22310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fae094e2a2540ab8b498f1985ac188d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fae094e2a2540ab8b498f1985ac188d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41053 } } }
I20260923 19:34:56.396157 22310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:56.396157 22312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:56.401569 22321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:56.407898 22321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:56.417944 22321 catalog_manager.cc:1347] Generated new cluster ID: 68bad597508240119a0902a4ef0e827d
I20260923 19:34:56.418099 22321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:56.428328 22321 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:56.430186 22321 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:56.449926 22321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fae094e2a2540ab8b498f1985ac188d: Generated new TSK 0
I20260923 19:34:56.451153 22321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:56.461027 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22233
W20260923 19:34:56.462698 22295 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34609
W20260923 19:34:56.463681 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22233: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:56.463838 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22233
W20260923 19:34:56.478318 22218 connection.cc:535] client connection to 127.0.97.62:41053 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:41053): Connection reset by peer (error 104)
2026-09-23T19:34:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:56 dist-test-slave-kxfb krb5kdc[22331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:56 dist-test-slave-kxfb krb5kdc[22331](info): set up 2 sockets
Sep 23 19:34:56 dist-test-slave-kxfb krb5kdc[22331](info): commencing operation
krb5kdc: starting...
W20260923 19:34:57.486690 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:57 dist-test-slave-kxfb krb5kdc[22331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:34:57.768837 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38711
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:58.091487 22344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:58.091794 22344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:58.091930 22344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:58.099221 22344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:58.099367 22344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:58.099445 22344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:58.099525 22344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:58.099586 22344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:34:58.110239 22344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:34:58.111577 22344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:34:58.112883 22344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:34:58.125666 22352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:58.125885 22349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:34:58.125736 22350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:34:58.127056 22344 server_base.cc:1034] running on GCE node
I20260923 19:34:58.127916 22344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:34:58.129424 22344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:34:58.130638 22344 hybrid_clock.cc:648] HybridClock initialized: now 1790192098130582 us; error 52 us; skew 500 ppm
I20260923 19:34:58.131016 22344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:34:58 dist-test-slave-kxfb krb5kdc[22331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:34:58.134212 22344 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:34:58.135948 22344 webserver.cc:460] Webserver started at http://127.0.97.62:36915/ using document root <none> and password file <none>
I20260923 19:34:58.136641 22344 fs_manager.cc:359] Metadata directory not provided
I20260923 19:34:58.136744 22344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:34:58.137051 22344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:34:58.140120 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "eb01d028ca5b4915b5e481f9f589d4ae"
format_stamp: "Formatted at 2026-09-23 19:34:58 on dist-test-slave-kxfb"
server_key: "b016bcc897ebe18c006cedface8748db"
server_key_iv: "d92ed29514e96fa3578ba86c0c47aa5a"
server_key_version: "encryptionkey@0"
I20260923 19:34:58.140887 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "eb01d028ca5b4915b5e481f9f589d4ae"
format_stamp: "Formatted at 2026-09-23 19:34:58 on dist-test-slave-kxfb"
server_key: "b016bcc897ebe18c006cedface8748db"
server_key_iv: "d92ed29514e96fa3578ba86c0c47aa5a"
server_key_version: "encryptionkey@0"
I20260923 19:34:58.146574 22344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 19:34:58.150403 22359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:34:58.152182 22344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:34:58.152375 22344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "eb01d028ca5b4915b5e481f9f589d4ae"
format_stamp: "Formatted at 2026-09-23 19:34:58 on dist-test-slave-kxfb"
server_key: "b016bcc897ebe18c006cedface8748db"
server_key_iv: "d92ed29514e96fa3578ba86c0c47aa5a"
server_key_version: "encryptionkey@0"
I20260923 19:34:58.152597 22344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:34:58.158594 22344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:34:58.162849 22344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:34:58.163120 22344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:34:58.181913 22344 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38711
I20260923 19:34:58.182030 22410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38711 every 8 connection(s)
I20260923 19:34:58.183501 22344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:34:58.186996 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22344
I20260923 19:34:58.187160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:34:58.187597 388 external_mini_cluster.cc:1428] Setting key 9a3c96e2bdc1cba62a46c7d0e4ad62f1
I20260923 19:34:58.188212 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:34:58.196974 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:58.189628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43423 (local address 127.0.97.62:38711)
0923 19:34:58.190601 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:58.190619 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:58.190680 (+ 61us) server_negotiation.cc:407] Connection header received
0923 19:34:58.192499 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:58.192529 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:58.193077 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:58.193615 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:58.194736 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:58.194745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:58.194763 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:58.194781 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:58.194875 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:58.194903 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:58.195168 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:58.195309 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 19:34:58.195543 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":389,"threads_started":1}
I20260923 19:34:58.200938 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:34:58.205175 22411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae: Bootstrap starting.
I20260923 19:34:58.209826 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:34:58.207016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45841 (local address 127.0.97.62:38711)
0923 19:34:58.207253 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 19:34:58.207264 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 19:34:58.207731 (+ 467us) server_negotiation.cc:407] Connection header received
0923 19:34:58.207958 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:34:58.207962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:34:58.208026 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:34:58.208115 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:34:58.209138 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:34:58.209143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:34:58.209146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:34:58.209163 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:34:58.209214 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:34:58.209232 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:34:58.209336 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 19:34:58.209455 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 19:34:58.209507 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 19:34:58.210050 22411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae: Neither blocks nor log segments found. Creating new log.
I20260923 19:34:58.211602 22411 log.cc:824] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae: Log is configured to *not* fsync() on all Append() calls
I20260923 19:34:58.215305 22411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae: No bootstrap required, opened a new log
I20260923 19:34:58.224097 22411 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb01d028ca5b4915b5e481f9f589d4ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38711 } }
I20260923 19:34:58.224601 22411 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:34:58.224710 22411 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb01d028ca5b4915b5e481f9f589d4ae, State: Initialized, Role: FOLLOWER
I20260923 19:34:58.225440 22411 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb01d028ca5b4915b5e481f9f589d4ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38711 } }
I20260923 19:34:58.225713 22411 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:34:58.225862 22411 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:34:58.226042 22411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:34:58.229012 22411 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb01d028ca5b4915b5e481f9f589d4ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38711 } }
I20260923 19:34:58.229645 22411 leader_election.cc:302] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb01d028ca5b4915b5e481f9f589d4ae; no voters:
I20260923 19:34:58.230345 22411 leader_election.cc:288] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:34:58.230549 22421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:34:58.231312 22421 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [term 1 LEADER]: Becoming Leader. State: Replica: eb01d028ca5b4915b5e481f9f589d4ae, State: Running, Role: LEADER
I20260923 19:34:58.231959 22421 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb01d028ca5b4915b5e481f9f589d4ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38711 } }
I20260923 19:34:58.232759 22411 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:34:58.234833 22422 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb01d028ca5b4915b5e481f9f589d4ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb01d028ca5b4915b5e481f9f589d4ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38711 } } }
I20260923 19:34:58.235388 22422 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:58.235361 22423 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb01d028ca5b4915b5e481f9f589d4ae. Latest consensus state: current_term: 1 leader_uuid: "eb01d028ca5b4915b5e481f9f589d4ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb01d028ca5b4915b5e481f9f589d4ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38711 } } }
I20260923 19:34:58.235649 22423 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae [sys.catalog]: This master's current role is: LEADER
I20260923 19:34:58.237112 22427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:34:58.246011 22427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:34:58.254909 22427 catalog_manager.cc:1347] Generated new cluster ID: 25f03dbbbcc849429ef3ba1430bf98ff
I20260923 19:34:58.255038 22427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:34:58.272754 22427 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:34:58.273516 22427 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:34:58.281040 22427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb01d028ca5b4915b5e481f9f589d4ae: Generated new TSK 0
I20260923 19:34:58.282421 22427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:34:58.292169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22344
W20260923 19:34:58.293498 22406 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43423
W20260923 19:34:58.294389 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22344: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:34:58.294621 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22344
2026-09-23T19:34:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1831 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:34:58 dist-test-slave-kxfb krb5kdc[22442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:34:58 dist-test-slave-kxfb krb5kdc[22442](info): set up 2 sockets
Sep 23 19:34:58 dist-test-slave-kxfb krb5kdc[22442](info): commencing operation
krb5kdc: starting...
W20260923 19:34:59.334534 388 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:34:59 dist-test-slave-kxfb krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:34:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:34:59.623734 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41893
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:34:59.983137 22455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:34:59.983419 22455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:34:59.983529 22455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:34:59.990538 22455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:34:59.990650 22455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:34:59.990698 22455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:34:59.990783 22455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:34:59.990859 22455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:35:00.000597 22455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:35:00.001929 22455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:35:00.003248 22455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:35:00.015458 22460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:00.015393 22463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:00.015432 22461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:35:00.016803 22455 server_base.cc:1034] running on GCE node
I20260923 19:35:00.017705 22455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:35:00.019431 22455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:35:00.020665 22455 hybrid_clock.cc:648] HybridClock initialized: now 1790192100020601 us; error 51 us; skew 500 ppm
I20260923 19:35:00.021083 22455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:35:00 dist-test-slave-kxfb krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:35:00.024657 22455 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:35:00.026613 22455 webserver.cc:460] Webserver started at http://127.0.97.62:42539/ using document root <none> and password file <none>
I20260923 19:35:00.027418 22455 fs_manager.cc:359] Metadata directory not provided
I20260923 19:35:00.027545 22455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:35:00.027865 22455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:35:00.031095 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "f801750f725b4d57b44ccb6359bf216c"
format_stamp: "Formatted at 2026-09-23 19:35:00 on dist-test-slave-kxfb"
server_key: "ad5de9133c89583b0b2f2ddac6568c6e"
server_key_iv: "6fb5e2ef97367c3697df5d9d7dcd5f1b"
server_key_version: "encryptionkey@0"
I20260923 19:35:00.031909 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "f801750f725b4d57b44ccb6359bf216c"
format_stamp: "Formatted at 2026-09-23 19:35:00 on dist-test-slave-kxfb"
server_key: "ad5de9133c89583b0b2f2ddac6568c6e"
server_key_iv: "6fb5e2ef97367c3697df5d9d7dcd5f1b"
server_key_version: "encryptionkey@0"
I20260923 19:35:00.037925 22455 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 19:35:00.042428 22470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:35:00.043980 22455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 19:35:00.044152 22455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "f801750f725b4d57b44ccb6359bf216c"
format_stamp: "Formatted at 2026-09-23 19:35:00 on dist-test-slave-kxfb"
server_key: "ad5de9133c89583b0b2f2ddac6568c6e"
server_key_iv: "6fb5e2ef97367c3697df5d9d7dcd5f1b"
server_key_version: "encryptionkey@0"
I20260923 19:35:00.044356 22455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:35:00.069036 22455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:35:00.073536 22455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:35:00.073784 22455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:35:00.093223 22455 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41893
I20260923 19:35:00.093241 22521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41893 every 8 connection(s)
I20260923 19:35:00.094635 22455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:35:00.099416 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:35:00.103549 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22455
I20260923 19:35:00.103766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:35:00.104388 388 external_mini_cluster.cc:1428] Setting key 8777c33916a37211210507f0ec7ca644
I20260923 19:35:00.113622 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c: Bootstrap starting.
I20260923 19:35:00.117467 22522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c: Neither blocks nor log segments found. Creating new log.
Sep 23 19:35:00 dist-test-slave-kxfb krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:35:00.118814 22522 log.cc:824] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c: Log is configured to *not* fsync() on all Append() calls
I20260923 19:35:00.122364 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c: No bootstrap required, opened a new log
I20260923 19:35:00.127552 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:00.106770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38433 (local address 127.0.97.62:41893)
0923 19:35:00.107685 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:00.107701 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:00.107745 (+ 44us) server_negotiation.cc:407] Connection header received
0923 19:35:00.109656 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:00.109681 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:00.110166 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:00.110580 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:00.111992 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:00.113321 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:00.114599 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:00.115290 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:00.115459 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:00.119900 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:00.119912 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:00.119927 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:00.119981 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:00.123174 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:00.123916 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:00.123922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:00.123930 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:00.124107 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:00.124483 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:00.124491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:00.124493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:00.125091 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:00.125407 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:00.125875 (+ 468us) server_negotiation.cc:299] Negotiation successful
0923 19:35:00.126394 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":399,"threads_started":1}
I20260923 19:35:00.131718 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:35:00.131747 22522 raft_consensus.cc:348] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f801750f725b4d57b44ccb6359bf216c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41893 } }
I20260923 19:35:00.132261 22522 raft_consensus.cc:374] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:35:00.132367 22522 raft_consensus.cc:729] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f801750f725b4d57b44ccb6359bf216c, State: Initialized, Role: FOLLOWER
I20260923 19:35:00.133105 22522 consensus_queue.cc:260] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f801750f725b4d57b44ccb6359bf216c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41893 } }
I20260923 19:35:00.133352 22522 raft_consensus.cc:388] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:35:00.133504 22522 raft_consensus.cc:482] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:35:00.133677 22522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:35:00.136370 22522 raft_consensus.cc:504] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f801750f725b4d57b44ccb6359bf216c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41893 } }
I20260923 19:35:00.137091 22522 leader_election.cc:302] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f801750f725b4d57b44ccb6359bf216c; no voters:
I20260923 19:35:00.137813 22522 leader_election.cc:288] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:35:00.138002 22532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:35:00.138859 22532 raft_consensus.cc:686] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [term 1 LEADER]: Becoming Leader. State: Replica: f801750f725b4d57b44ccb6359bf216c, State: Running, Role: LEADER
I20260923 19:35:00.139632 22532 consensus_queue.cc:237] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f801750f725b4d57b44ccb6359bf216c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41893 } }
I20260923 19:35:00.141971 22532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f801750f725b4d57b44ccb6359bf216c. Latest consensus state: current_term: 1 leader_uuid: "f801750f725b4d57b44ccb6359bf216c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f801750f725b4d57b44ccb6359bf216c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41893 } } }
I20260923 19:35:00.142409 22532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:00.142901 22534 sys_catalog.cc:455] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f801750f725b4d57b44ccb6359bf216c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f801750f725b4d57b44ccb6359bf216c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41893 } } }
I20260923 19:35:00.143321 22534 sys_catalog.cc:458] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:00.146943 22522 sys_catalog.cc:564] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:35:00.151093 22536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:35:00.153924 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:00.136906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52241 (local address 127.0.97.62:41893)
0923 19:35:00.137184 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:00.137193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:00.137681 (+ 488us) server_negotiation.cc:407] Connection header received
0923 19:35:00.137881 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:00.137887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:00.138008 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:00.138175 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:00.140204 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:00.141415 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:00.143291 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:00.144031 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:00.144169 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:00.147600 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:00.147609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:00.147614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:00.147665 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:00.151225 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:00.151904 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:00.151909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:00.151916 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:00.152053 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:00.152403 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:00.152407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:00.152409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:00.152753 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:00.152986 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:00.153330 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 19:35:00.153557 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 19:35:00.158039 22536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:35:00.173774 22536 catalog_manager.cc:1347] Generated new cluster ID: cadbd7888619477a8a05ce83b2f19527
I20260923 19:35:00.173959 22536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:35:00.186369 22536 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:35:00.188537 22536 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:35:00.195851 22536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f801750f725b4d57b44ccb6359bf216c: Generated new TSK 0
I20260923 19:35:00.197204 22536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:35:00.204345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22455
I20260923 19:35:00.205595 22517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:35:00.303435 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22455
2026-09-23T19:35:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:35:00 dist-test-slave-kxfb krb5kdc[22554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:35:00 dist-test-slave-kxfb krb5kdc[22554](info): set up 2 sockets
Sep 23 19:35:00 dist-test-slave-kxfb krb5kdc[22554](info): commencing operation
krb5kdc: starting...
W20260923 19:35:01.346967 388 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:35:01 dist-test-slave-kxfb krb5kdc[22554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:35:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:35:01.640235 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40325
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:35:02.011049 22567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:35:02.011359 22567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:35:02.011452 22567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:35:02.018781 22567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:35:02.018906 22567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:35:02.018952 22567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:35:02.019040 22567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:35:02.019106 22567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:35:02.029604 22567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:35:02.031042 22567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:35:02.032373 22567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:35:02.045475 22575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:02.045377 22573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:02.045785 22572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:35:02.046607 22567 server_base.cc:1034] running on GCE node
I20260923 19:35:02.047465 22567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:35:02.049039 22567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:35:02.050256 22567 hybrid_clock.cc:648] HybridClock initialized: now 1790192102050196 us; error 59 us; skew 500 ppm
I20260923 19:35:02.050635 22567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:35:02 dist-test-slave-kxfb krb5kdc[22554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:35:02.057541 22567 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:35:02.059409 22567 webserver.cc:460] Webserver started at http://127.0.97.62:45925/ using document root <none> and password file <none>
I20260923 19:35:02.060170 22567 fs_manager.cc:359] Metadata directory not provided
I20260923 19:35:02.060297 22567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:35:02.060598 22567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:35:02.063874 22567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "ce94aacca9e64743b5d82d8db51cf6ce"
format_stamp: "Formatted at 2026-09-23 19:35:02 on dist-test-slave-kxfb"
server_key: "e722c21376fd1024117e5f8ddf434234"
server_key_iv: "27e7b18586e030d60e339e88680a8e7c"
server_key_version: "encryptionkey@0"
I20260923 19:35:02.064651 22567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "ce94aacca9e64743b5d82d8db51cf6ce"
format_stamp: "Formatted at 2026-09-23 19:35:02 on dist-test-slave-kxfb"
server_key: "e722c21376fd1024117e5f8ddf434234"
server_key_iv: "27e7b18586e030d60e339e88680a8e7c"
server_key_version: "encryptionkey@0"
I20260923 19:35:02.070320 22567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:35:02.074191 22582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:35:02.075786 22567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 19:35:02.075960 22567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "ce94aacca9e64743b5d82d8db51cf6ce"
format_stamp: "Formatted at 2026-09-23 19:35:02 on dist-test-slave-kxfb"
server_key: "e722c21376fd1024117e5f8ddf434234"
server_key_iv: "27e7b18586e030d60e339e88680a8e7c"
server_key_version: "encryptionkey@0"
I20260923 19:35:02.076145 22567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:35:02.102406 22567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:35:02.106609 22567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:35:02.106878 22567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:35:02.126242 22567 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40325
I20260923 19:35:02.126341 22633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40325 every 8 connection(s)
I20260923 19:35:02.127702 22567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:35:02.129688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22567
I20260923 19:35:02.129863 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:35:02.130357 388 external_mini_cluster.cc:1428] Setting key cd08e8395cd73a0e3b5475a7f569681e
I20260923 19:35:02.133258 22634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:35:02.143854 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:02.132596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41497 (local address 127.0.97.62:40325)
0923 19:35:02.133516 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:02.133531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:02.133577 (+ 46us) server_negotiation.cc:407] Connection header received
0923 19:35:02.135356 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:02.135382 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:02.135847 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:02.136267 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:02.136837 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:02.138167 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:02.139447 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:02.140113 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:02.140278 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:02.141431 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:02.141440 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:02.141454 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:35:02.142050 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:02.142124 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:35:02.142167 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:02.142278 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:02.142519 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 19:35:02.142791 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":382,"threads_started":1}
I20260923 19:35:02.145965 22634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce: Bootstrap starting.
I20260923 19:35:02.147706 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:35:02.150327 22634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce: Neither blocks nor log segments found. Creating new log.
I20260923 19:35:02.151907 22634 log.cc:824] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce: Log is configured to *not* fsync() on all Append() calls
I20260923 19:35:02.155898 22634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce: No bootstrap required, opened a new log
I20260923 19:35:02.160701 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:02.153508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44811 (local address 127.0.97.62:40325)
0923 19:35:02.153737 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:02.153745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:02.154238 (+ 493us) server_negotiation.cc:407] Connection header received
0923 19:35:02.154463 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:02.154467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:02.154529 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:02.154614 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:02.155330 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:02.156069 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:02.157959 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:02.158567 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:02.158680 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:02.159935 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:02.159940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:02.159943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:35:02.159960 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:02.160006 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:35:02.160021 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:02.160100 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:02.160252 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 19:35:02.160327 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 19:35:02.165750 22634 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94aacca9e64743b5d82d8db51cf6ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } }
I20260923 19:35:02.166232 22634 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:35:02.166354 22634 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce94aacca9e64743b5d82d8db51cf6ce, State: Initialized, Role: FOLLOWER
I20260923 19:35:02.167057 22634 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94aacca9e64743b5d82d8db51cf6ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } }
I20260923 19:35:02.167290 22634 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:35:02.167434 22634 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:35:02.167606 22634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:35:02.170574 22634 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94aacca9e64743b5d82d8db51cf6ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } }
I20260923 19:35:02.171193 22634 leader_election.cc:302] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce94aacca9e64743b5d82d8db51cf6ce; no voters:
I20260923 19:35:02.171954 22634 leader_election.cc:288] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:35:02.172094 22644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:35:02.173209 22644 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [term 1 LEADER]: Becoming Leader. State: Replica: ce94aacca9e64743b5d82d8db51cf6ce, State: Running, Role: LEADER
I20260923 19:35:02.173818 22644 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94aacca9e64743b5d82d8db51cf6ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } }
I20260923 19:35:02.174418 22634 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:35:02.177379 22646 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce94aacca9e64743b5d82d8db51cf6ce. Latest consensus state: current_term: 1 leader_uuid: "ce94aacca9e64743b5d82d8db51cf6ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94aacca9e64743b5d82d8db51cf6ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } } }
I20260923 19:35:02.177436 22645 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce94aacca9e64743b5d82d8db51cf6ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce94aacca9e64743b5d82d8db51cf6ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40325 } } }
I20260923 19:35:02.177829 22646 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:02.177876 22645 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:02.179420 22653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:35:02.185070 22653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:35:02.194768 22653 catalog_manager.cc:1347] Generated new cluster ID: 3a6f99b0d2e94a9db9d3e8e7d5f9c188
I20260923 19:35:02.194883 22653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:35:02.209256 22653 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:35:02.210695 22653 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:35:02.223202 22653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce94aacca9e64743b5d82d8db51cf6ce: Generated new TSK 0
I20260923 19:35:02.224155 22653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:35:02.231258 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22567
W20260923 19:35:02.232698 22629 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41497
W20260923 19:35:02.233421 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22567: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:35:02.233655 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22567
2026-09-23T19:35:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:35:02 dist-test-slave-kxfb krb5kdc[22665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:35:02 dist-test-slave-kxfb krb5kdc[22665](info): set up 2 sockets
Sep 23 19:35:02 dist-test-slave-kxfb krb5kdc[22665](info): commencing operation
krb5kdc: starting...
W20260923 19:35:03.275147 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:35:03 dist-test-slave-kxfb krb5kdc[22665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:35:03.569607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34093
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:35:03.955680 22678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:35:03.955999 22678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:35:03.956095 22678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:35:03.963351 22678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:35:03.963471 22678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:35:03.963517 22678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:35:03.963588 22678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:35:03.963658 22678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:35:03.973585 22678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:35:03.974980 22678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:35:03.976316 22678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:35:03.990199 22686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:03.990224 22683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:03.990137 22684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:35:03.990935 22678 server_base.cc:1034] running on GCE node
I20260923 19:35:03.992044 22678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:35:03.993690 22678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:35:03.994911 22678 hybrid_clock.cc:648] HybridClock initialized: now 1790192103994855 us; error 49 us; skew 500 ppm
I20260923 19:35:03.995308 22678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:35:03 dist-test-slave-kxfb krb5kdc[22665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:35:03.998816 22678 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:35:04.000625 22678 webserver.cc:460] Webserver started at http://127.0.97.62:39959/ using document root <none> and password file <none>
I20260923 19:35:04.001372 22678 fs_manager.cc:359] Metadata directory not provided
I20260923 19:35:04.001482 22678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:35:04.001803 22678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:35:04.004922 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "2190c0e6c25e4bfb9b06c1d00598253c"
format_stamp: "Formatted at 2026-09-23 19:35:03 on dist-test-slave-kxfb"
server_key: "87ce3cd59cd7307a647d902574d8eae3"
server_key_iv: "676b0476d93aad5d6aba2efe578ce4bd"
server_key_version: "encryptionkey@0"
I20260923 19:35:04.005787 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "2190c0e6c25e4bfb9b06c1d00598253c"
format_stamp: "Formatted at 2026-09-23 19:35:03 on dist-test-slave-kxfb"
server_key: "87ce3cd59cd7307a647d902574d8eae3"
server_key_iv: "676b0476d93aad5d6aba2efe578ce4bd"
server_key_version: "encryptionkey@0"
I20260923 19:35:04.011605 22678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 19:35:04.015540 22693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:35:04.017087 22678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 19:35:04.017292 22678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "2190c0e6c25e4bfb9b06c1d00598253c"
format_stamp: "Formatted at 2026-09-23 19:35:03 on dist-test-slave-kxfb"
server_key: "87ce3cd59cd7307a647d902574d8eae3"
server_key_iv: "676b0476d93aad5d6aba2efe578ce4bd"
server_key_version: "encryptionkey@0"
I20260923 19:35:04.017499 22678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:35:04.023860 22678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:35:04.028532 22678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:35:04.028815 22678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:35:04.048686 22678 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34093
I20260923 19:35:04.048780 22744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34093 every 8 connection(s)
I20260923 19:35:04.050186 22678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:35:04.051252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22678
I20260923 19:35:04.051473 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:35:04.052069 388 external_mini_cluster.cc:1428] Setting key ade416ffb6fd1a504e57ba0f5ef2c0c9
I20260923 19:35:04.055990 22745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:35:04.060799 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:04.054273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50933 (local address 127.0.97.62:34093)
0923 19:35:04.055211 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:04.055227 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:04.055278 (+ 51us) server_negotiation.cc:407] Connection header received
0923 19:35:04.057103 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:04.057129 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:04.057558 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:04.057963 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:04.059100 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:04.059107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:04.059119 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:35:04.059134 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:04.059233 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:35:04.059256 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:04.059486 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:04.059643 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 19:35:04.059830 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":366,"threads_started":1}
I20260923 19:35:04.064705 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:35:04.069545 22745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c: Bootstrap starting.
I20260923 19:35:04.071882 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:04.069041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36269 (local address 127.0.97.62:34093)
0923 19:35:04.069244 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:04.069249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:04.069690 (+ 441us) server_negotiation.cc:407] Connection header received
0923 19:35:04.069960 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:04.069964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:04.070023 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:04.070120 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:04.071180 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:04.071184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:04.071187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 19:35:04.071202 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:04.071244 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 19:35:04.071259 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:04.071362 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:04.071518 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 19:35:04.071573 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 19:35:04.073953 22745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c: Neither blocks nor log segments found. Creating new log.
I20260923 19:35:04.075206 22745 log.cc:824] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c: Log is configured to *not* fsync() on all Append() calls
I20260923 19:35:04.078505 22745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c: No bootstrap required, opened a new log
I20260923 19:35:04.086360 22745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190c0e6c25e4bfb9b06c1d00598253c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } }
I20260923 19:35:04.086809 22745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 19:35:04.086908 22745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2190c0e6c25e4bfb9b06c1d00598253c, State: Initialized, Role: FOLLOWER
I20260923 19:35:04.087589 22745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190c0e6c25e4bfb9b06c1d00598253c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } }
I20260923 19:35:04.087827 22745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 19:35:04.087973 22745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 19:35:04.088143 22745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:35:04.090557 22745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190c0e6c25e4bfb9b06c1d00598253c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } }
I20260923 19:35:04.091145 22745 leader_election.cc:302] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2190c0e6c25e4bfb9b06c1d00598253c; no voters:
I20260923 19:35:04.091817 22745 leader_election.cc:288] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 19:35:04.092012 22755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:35:04.092770 22755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [term 1 LEADER]: Becoming Leader. State: Replica: 2190c0e6c25e4bfb9b06c1d00598253c, State: Running, Role: LEADER
I20260923 19:35:04.093457 22755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190c0e6c25e4bfb9b06c1d00598253c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } }
I20260923 19:35:04.094349 22745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:35:04.096707 22756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2190c0e6c25e4bfb9b06c1d00598253c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190c0e6c25e4bfb9b06c1d00598253c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } } }
I20260923 19:35:04.096690 22757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2190c0e6c25e4bfb9b06c1d00598253c. Latest consensus state: current_term: 1 leader_uuid: "2190c0e6c25e4bfb9b06c1d00598253c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190c0e6c25e4bfb9b06c1d00598253c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34093 } } }
I20260923 19:35:04.097381 22756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:04.099448 22757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:04.100111 22763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:35:04.105683 22763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:35:04.117043 22763 catalog_manager.cc:1347] Generated new cluster ID: 715dfbb754314918847a4e0ebad853d3
I20260923 19:35:04.117177 22763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:35:04.148753 22763 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:35:04.149327 22763 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:35:04.161062 22763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2190c0e6c25e4bfb9b06c1d00598253c: Generated new TSK 0
I20260923 19:35:04.162256 22763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 19:35:04.181547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22678
W20260923 19:35:04.182976 22740 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50933
W20260923 19:35:04.184008 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22678: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 19:35:04.184171 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22678
2026-09-23T19:35:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1950 ms)
[----------] 18 tests from AuthTokenIssuingTest (34977 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:35:04 dist-test-slave-kxfb krb5kdc[22776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:35:04 dist-test-slave-kxfb krb5kdc[22776](info): set up 2 sockets
Sep 23 19:35:04 dist-test-slave-kxfb krb5kdc[22776](info): commencing operation
krb5kdc: starting...
W20260923 19:35:05.227443 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:35:05 dist-test-slave-kxfb krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:35:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:35:05.534474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35665
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35665,127.0.97.61:37947,127.0.97.60:42947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:35:05.912106 22789 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 19:35:05.912525 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:35:05.912638 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:35:05.912760 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:35:05.920511 22789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:35:05.920617 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:35:05.920689 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:35:05.920768 22789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:35:05.920851 22789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:35:05.931807 22789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35665,127.0.97.61:37947,127.0.97.60:42947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:35:05.933498 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:35:05.935066 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:35:05.948174 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:05.948267 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:05.948269 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:35:05.949213 22789 server_base.cc:1034] running on GCE node
I20260923 19:35:05.950151 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:35:05.951927 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:35:05.953162 22789 hybrid_clock.cc:648] HybridClock initialized: now 1790192105953097 us; error 43 us; skew 500 ppm
I20260923 19:35:05.953560 22789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:35:05 dist-test-slave-kxfb krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:35:05.957050 22789 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:35:05.958811 22789 webserver.cc:460] Webserver started at http://127.0.97.62:35247/ using document root <none> and password file <none>
I20260923 19:35:05.959654 22789 fs_manager.cc:359] Metadata directory not provided
I20260923 19:35:05.959766 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:35:05.960116 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:35:05.963704 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "c65817382bd54c9a9ab7cdc3cc0400c6"
format_stamp: "Formatted at 2026-09-23 19:35:05 on dist-test-slave-kxfb"
server_key: "99694cd90891931fa00cfa06f8a8463e"
server_key_iv: "94086590f8aca4a9cff7b3689b327d74"
server_key_version: "encryptionkey@0"
I20260923 19:35:05.964592 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "c65817382bd54c9a9ab7cdc3cc0400c6"
format_stamp: "Formatted at 2026-09-23 19:35:05 on dist-test-slave-kxfb"
server_key: "99694cd90891931fa00cfa06f8a8463e"
server_key_iv: "94086590f8aca4a9cff7b3689b327d74"
server_key_version: "encryptionkey@0"
I20260923 19:35:05.970498 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 19:35:05.974365 22804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:35:05.975871 22789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 19:35:05.976047 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "c65817382bd54c9a9ab7cdc3cc0400c6"
format_stamp: "Formatted at 2026-09-23 19:35:05 on dist-test-slave-kxfb"
server_key: "99694cd90891931fa00cfa06f8a8463e"
server_key_iv: "94086590f8aca4a9cff7b3689b327d74"
server_key_version: "encryptionkey@0"
I20260923 19:35:05.976284 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:35:06.009148 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:35:06.014130 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:35:06.014463 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:35:06.034795 22789 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35665
I20260923 19:35:06.034834 22855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35665 every 8 connection(s)
I20260923 19:35:06.036535 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:35:06.040745 22856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:35:06.046829 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22789
I20260923 19:35:06.047091 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:35:06.047820 388 external_mini_cluster.cc:1428] Setting key b34366f322bbb9358a26d02cd2826c14
I20260923 19:35:06.047895 22856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:06 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:35:06.070218 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.051165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.0.97.62:35665)
0923 19:35:06.052890 (+ 1725us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:06.052899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:06.052952 (+ 53us) server_negotiation.cc:407] Connection header received
0923 19:35:06.054569 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:06.054575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:06.054808 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:06.054972 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:06.056237 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.057431 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.058797 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.059318 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.059485 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:06.063208 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:06.063216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:06.063229 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:06.063257 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:06.065946 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:06.066671 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:06.066678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:06.066680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:06.066819 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:06.067185 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:06.067188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:06.067190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:06.067692 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:06.067876 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:06.068410 (+ 534us) server_negotiation.cc:299] Negotiation successful
0923 19:35:06.068785 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":1515,"thread_start_us":1256,"threads_started":1}
I20260923 19:35:06.070250 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.051157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35665 (local address 127.0.0.1:54694)
0923 19:35:06.052088 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.052189 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:06.054189 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:06.055012 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:06.055019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:06.055719 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:06.056044 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:06.056060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.057575 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.057579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:06.058673 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:06.058686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.059536 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.059543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:06.059707 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:06.060450 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:06.060471 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:06.062967 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:06.066089 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:06.066100 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:06.066112 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:06.066541 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:06.066928 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:06.066932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:06.066934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:06.067077 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:06.067820 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:06.067827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:06.068038 (+ 211us) client_negotiation.cc:769] Sending connection context
0923 19:35:06.068267 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 19:35:06.068908 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":172,"thread_start_us":380,"threads_started":1}
I20260923 19:35:06.073774 22856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:35:06.074960 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.074564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59274)
0923 19:35:06.074744 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.074854 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 19:35:06.075212 22808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260923 19:35:06.076669 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.076331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59278)
0923 19:35:06.076460 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.076539 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 19:35:06.076893 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260923 19:35:06.113109 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37947
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35665,127.0.97.61:37947,127.0.97.60:42947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:35:06.126246 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } attempt: 1
I20260923 19:35:06.127724 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.127190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59292)
0923 19:35:06.127442 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.127555 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 19:35:06.128985 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.128557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59308)
0923 19:35:06.128709 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.128806 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 19:35:06.129297 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
I20260923 19:35:06.197613 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } attempt: 2
I20260923 19:35:06.199064 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.198551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59314)
0923 19:35:06.198792 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.198916 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 19:35:06.200133 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.199769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59318)
0923 19:35:06.199907 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.199993 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 19:35:06.200309 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
I20260923 19:35:06.291596 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } attempt: 3
I20260923 19:35:06.292709 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.292288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59326)
0923 19:35:06.292487 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.292569 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 19:35:06.293587 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.293261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59342)
0923 19:35:06.293396 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.293457 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 19:35:06.293751 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
I20260923 19:35:06.437013 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } attempt: 4
I20260923 19:35:06.438201 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.437692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59352)
0923 19:35:06.437993 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.438073 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 19:35:06.439462 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.438818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59356)
0923 19:35:06.439086 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.439225 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260923 19:35:06.439702 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37947: connect: Connection refused (error 111)
W20260923 19:35:06.495906 22863 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 19:35:06.496340 22863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:35:06.496452 22863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:35:06.496575 22863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:35:06.504683 22863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:35:06.504808 22863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:35:06.504878 22863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:35:06.504963 22863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:35:06.505052 22863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:35:06.515990 22863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35665,127.0.97.61:37947,127.0.97.60:42947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:35:06.517679 22863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:35:06.519258 22863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:35:06.532634 22871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:06.532651 22869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:06.533234 22868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:35:06.534263 22863 server_base.cc:1034] running on GCE node
I20260923 19:35:06.535270 22863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:35:06.537091 22863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:35:06.538373 22863 hybrid_clock.cc:648] HybridClock initialized: now 1790192106538264 us; error 60 us; skew 500 ppm
I20260923 19:35:06.538781 22863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:35:06 dist-test-slave-kxfb krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:35:06.542459 22863 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260923 19:35:06.544344 22863 webserver.cc:460] Webserver started at http://127.0.97.61:46467/ using document root <none> and password file <none>
I20260923 19:35:06.545166 22863 fs_manager.cc:359] Metadata directory not provided
I20260923 19:35:06.545261 22863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:35:06.545552 22863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:35:06.548892 22863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/data/instance:
uuid: "9b1bc24539a2475baaa7086c9e7c1fcd"
format_stamp: "Formatted at 2026-09-23 19:35:06 on dist-test-slave-kxfb"
server_key: "6050cf094d6378ce3acf843b69a2b056"
server_key_iv: "01d38fcdb5b6a43070ca6ae762f0bfba"
server_key_version: "encryptionkey@0"
I20260923 19:35:06.549688 22863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/wal/instance:
uuid: "9b1bc24539a2475baaa7086c9e7c1fcd"
format_stamp: "Formatted at 2026-09-23 19:35:06 on dist-test-slave-kxfb"
server_key: "6050cf094d6378ce3acf843b69a2b056"
server_key_iv: "01d38fcdb5b6a43070ca6ae762f0bfba"
server_key_version: "encryptionkey@0"
I20260923 19:35:06.555622 22863 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 19:35:06.559836 22878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:35:06.561641 22863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 19:35:06.561856 22863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/wal
uuid: "9b1bc24539a2475baaa7086c9e7c1fcd"
format_stamp: "Formatted at 2026-09-23 19:35:06 on dist-test-slave-kxfb"
server_key: "6050cf094d6378ce3acf843b69a2b056"
server_key_iv: "01d38fcdb5b6a43070ca6ae762f0bfba"
server_key_version: "encryptionkey@0"
I20260923 19:35:06.562103 22863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:35:06.580438 22863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:35:06.585346 22863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:35:06.585598 22863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:35:06.605851 22863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37947
I20260923 19:35:06.605895 22929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37947 every 8 connection(s)
I20260923 19:35:06.607401 22863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
I20260923 19:35:06.611663 22930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:35:06.616043 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22863
I20260923 19:35:06.616230 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-1/wal/instance
I20260923 19:35:06.616694 388 external_mini_cluster.cc:1428] Setting key 4a7ae523674952e410e5ae1143889a7c
I20260923 19:35:06.618468 22930 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:06 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260923 19:35:06.649158 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.621305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.0.97.62:35665)
0923 19:35:06.630584 (+ 9279us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:06.630592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:06.631117 (+ 525us) server_negotiation.cc:407] Connection header received
0923 19:35:06.633463 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:06.633469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:06.633562 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:06.633714 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:06.635703 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.636722 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.638471 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.639069 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.639185 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:06.643147 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:06.643152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:06.643155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:06.643203 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:06.645800 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:06.646755 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:06.646760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:06.646762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:06.646891 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:06.647304 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:06.647310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:06.647313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:06.647615 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:06.647742 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:06.648562 (+ 820us) server_negotiation.cc:299] Negotiation successful
0923 19:35:06.648766 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9141,"thread_start_us":9018,"threads_started":1}
I20260923 19:35:06.650484 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.622105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35665 (local address 127.0.0.1:54710)
0923 19:35:06.630867 (+ 8762us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.630929 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:06.633117 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:06.634014 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:06.634023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:06.634758 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:06.635474 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:06.635494 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.636911 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.636916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:06.638261 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:06.638274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.639376 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.639382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:06.639628 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:06.640395 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:06.640430 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:06.642860 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:06.645971 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:06.645981 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:06.646000 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:06.646635 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:06.646982 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:06.646985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:06.646987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:06.647160 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:06.647734 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:06.647740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:06.648146 (+ 406us) client_negotiation.cc:769] Sending connection context
0923 19:35:06.648405 (+ 259us) client_negotiation.cc:240] Negotiation successful
0923 19:35:06.648862 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8398,"thread_start_us":8084,"threads_started":1}
I20260923 19:35:06.652522 22930 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:06 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:35:06.669847 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.653636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.0.97.61:37947)
0923 19:35:06.654406 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:06.654413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:06.654439 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:35:06.654496 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:06.654502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:06.654711 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:06.654854 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:06.655746 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.656828 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.658381 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.658942 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.659047 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:06.662967 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:06.662980 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:06.662994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:06.663030 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:06.666500 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:06.667304 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:06.667313 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:06.667316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:06.667447 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:06.668028 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:06.668033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:06.668035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:06.668478 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:06.668671 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:06.669168 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 19:35:06.669424 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":418,"threads_started":1}
I20260923 19:35:06.670763 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.653291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59366)
0923 19:35:06.653518 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.653541 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:06.653656 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:06.654889 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:06.654901 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:06.655373 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:06.655575 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:06.655585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.657008 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.657013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:06.658191 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:06.658204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.659098 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.659103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:06.659294 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:06.659935 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:06.659955 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:06.662718 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:06.666686 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:06.666691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:06.666695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:06.667179 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:06.667592 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:06.667596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:06.667600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:06.667909 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:06.668749 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:06.668753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:06.668885 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 19:35:06.670159 (+ 1274us) client_negotiation.cc:240] Negotiation successful
0923 19:35:06.670464 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 19:35:06.678663 22930 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:35:06.680260 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.679521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57790)
0923 19:35:06.679826 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.680028 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260923 19:35:06.680534 22882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
I20260923 19:35:06.682723 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.682228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57800)
0923 19:35:06.682442 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.682553 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260923 19:35:06.683156 22930 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260923 19:35:06.693516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42947
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35665,127.0.97.61:37947,127.0.97.60:42947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:35:06.732487 22930 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 1
I20260923 19:35:06.733963 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.733359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57814)
0923 19:35:06.733624 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.733756 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260923 19:35:06.735641 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.735204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57822)
0923 19:35:06.735389 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.735509 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 19:35:06.735863 22930 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
I20260923 19:35:06.738947 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } attempt: 5
Sep 23 19:35:06 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:35:06.754436 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.739623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59376)
0923 19:35:06.739889 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.739924 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:06.740104 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:06.740544 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:06.740550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:06.741125 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:06.741403 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:06.741416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.742990 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.742997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:06.744114 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:06.744129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.745189 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.745193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:06.745343 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:06.745953 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:06.745971 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:06.748037 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:06.752005 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:06.752009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:06.752012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:06.752396 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:06.752834 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:06.752837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:06.752840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:06.752968 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:06.753639 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:06.753642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:06.753779 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 19:35:06.753918 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 19:35:06.754132 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 19:35:06.754786 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.739708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.0.97.61:37947)
0923 19:35:06.739891 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:06.739897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:06.740054 (+ 157us) server_negotiation.cc:407] Connection header received
0923 19:35:06.740249 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:06.740255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:06.740362 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:06.740471 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:06.741600 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.742776 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.744296 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:06.745027 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:06.745158 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:06.748256 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:06.748262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:06.748267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:06.748310 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:06.751844 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:06.752527 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:06.752533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:06.752537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:06.752718 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:06.753077 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:06.753082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:06.753086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:06.753519 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:06.753703 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:06.754074 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 19:35:06.754397 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 19:35:06.755864 22856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:35:06.756768 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.756414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57838)
0923 19:35:06.756570 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.756641 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 19:35:06.757531 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.757254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57848)
0923 19:35:06.757378 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.757434 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 19:35:06.757709 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
I20260923 19:35:06.804131 22930 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 2
I20260923 19:35:06.805613 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.805013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57852)
0923 19:35:06.805261 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.805408 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260923 19:35:06.806677 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.806329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57864)
0923 19:35:06.806478 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.806553 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 19:35:06.806895 22930 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
I20260923 19:35:06.808917 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 1
I20260923 19:35:06.809890 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.809480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57878)
0923 19:35:06.809651 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.809736 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 19:35:06.810730 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.810451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57892)
0923 19:35:06.810579 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.810638 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 19:35:06.810904 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
I20260923 19:35:06.879148 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 2
I20260923 19:35:06.880652 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.880051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57896)
0923 19:35:06.880343 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.880454 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 19:35:06.881785 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.881470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57898)
0923 19:35:06.881609 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.881677 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 19:35:06.882028 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
I20260923 19:35:06.898154 22930 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 3
I20260923 19:35:06.899662 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.899072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57910)
0923 19:35:06.899356 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.899467 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260923 19:35:06.900909 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.900401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57920)
0923 19:35:06.900704 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.900786 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
W20260923 19:35:06.901157 22930 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
I20260923 19:35:06.988297 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 3
I20260923 19:35:06.989711 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.989153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57922)
0923 19:35:06.989413 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.989520 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 19:35:06.990728 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:06.990401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57930)
0923 19:35:06.990568 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:06.990632 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 19:35:06.990921 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
I20260923 19:35:07.044373 22930 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 4
I20260923 19:35:07.045720 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.045179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57942)
0923 19:35:07.045444 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:07.045556 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 19:35:07.046958 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.046417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57956)
0923 19:35:07.046640 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:07.046762 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 19:35:07.047219 22930 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
W20260923 19:35:07.070505 22938 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 19:35:07.070832 22938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:35:07.070902 22938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:35:07.070986 22938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:35:07.078313 22938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:35:07.078433 22938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:35:07.078491 22938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:35:07.078550 22938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:35:07.078622 22938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:35:07.088871 22938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35665,127.0.97.61:37947,127.0.97.60:42947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:35:07.090268 22938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:35:07.091703 22938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:35:07.103982 22943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:07.104096 22944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:07.104746 22946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:35:07.105559 22938 server_base.cc:1034] running on GCE node
I20260923 19:35:07.106495 22938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:35:07.108047 22938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:35:07.109318 22938 hybrid_clock.cc:648] HybridClock initialized: now 1790192107109235 us; error 55 us; skew 500 ppm
I20260923 19:35:07.109714 22938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:35:07 dist-test-slave-kxfb krb5kdc[22776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:35:07.116545 22938 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260923 19:35:07.118251 22938 webserver.cc:460] Webserver started at http://127.0.97.60:34611/ using document root <none> and password file <none>
I20260923 19:35:07.119032 22938 fs_manager.cc:359] Metadata directory not provided
I20260923 19:35:07.119134 22938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:35:07.119443 22938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:35:07.122483 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/data/instance:
uuid: "ae851d92d0814d70badcb3bb55bd1a67"
format_stamp: "Formatted at 2026-09-23 19:35:07 on dist-test-slave-kxfb"
server_key: "6ae0a3d39336ca5e1b8f76d5b8d6fd2d"
server_key_iv: "d046d82a266172c2461ae838aa49139a"
server_key_version: "encryptionkey@0"
I20260923 19:35:07.123263 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/wal/instance:
uuid: "ae851d92d0814d70badcb3bb55bd1a67"
format_stamp: "Formatted at 2026-09-23 19:35:07 on dist-test-slave-kxfb"
server_key: "6ae0a3d39336ca5e1b8f76d5b8d6fd2d"
server_key_iv: "d046d82a266172c2461ae838aa49139a"
server_key_version: "encryptionkey@0"
I20260923 19:35:07.128793 22938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 19:35:07.132441 22953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:35:07.133914 22938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 19:35:07.134079 22938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/wal
uuid: "ae851d92d0814d70badcb3bb55bd1a67"
format_stamp: "Formatted at 2026-09-23 19:35:07 on dist-test-slave-kxfb"
server_key: "6ae0a3d39336ca5e1b8f76d5b8d6fd2d"
server_key_iv: "d046d82a266172c2461ae838aa49139a"
server_key_version: "encryptionkey@0"
I20260923 19:35:07.134325 22938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:35:07.159021 22938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:35:07.163148 22938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:35:07.163401 22938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:35:07.168134 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 4
I20260923 19:35:07.169562 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.168949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57966)
0923 19:35:07.169242 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:07.169369 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 19:35:07.170900 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.170454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:57982)
0923 19:35:07.170658 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:07.170761 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 19:35:07.171133 22856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42947: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42947: connect: Connection refused (error 111)
I20260923 19:35:07.182960 22938 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42947
I20260923 19:35:07.183009 23004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42947 every 8 connection(s)
I20260923 19:35:07.184332 22938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
I20260923 19:35:07.188841 23005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:35:07.193754 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 22938
I20260923 19:35:07.193917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790191744267121-388-0/minicluster-data/master-2/wal/instance
I20260923 19:35:07.194396 388 external_mini_cluster.cc:1428] Setting key 40ca89f9b91ce07431a55cff92fcd707
I20260923 19:35:07.196993 23005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:07 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 19:35:07 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:35:07.218991 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.202111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.0.97.62:35665)
0923 19:35:07.203676 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.203680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.203699 (+ 19us) server_negotiation.cc:407] Connection header received
0923 19:35:07.203757 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.203760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.203820 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.203916 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.205339 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.206436 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.208201 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.208815 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.208960 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.212916 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.212922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.212926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.212969 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.215832 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.216531 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.216536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.216538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.216687 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.217087 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.217091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.217093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.217375 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.217497 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.218320 (+ 823us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.218559 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466,"thread_start_us":1317,"threads_started":1}
I20260923 19:35:07.220525 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.201919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35665 (local address 127.0.0.1:54724)
0923 19:35:07.203021 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:07.203078 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:07.203268 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:07.204028 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:07.204036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:07.204858 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:07.205138 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.205147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.206595 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.206600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.208041 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.208053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.209048 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.209053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.209182 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.209838 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:07.209862 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:07.212640 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:07.215954 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.215962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.215975 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.216389 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.216797 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.216802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.216806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.216976 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.217500 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:07.217509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:07.217853 (+ 344us) client_negotiation.cc:769] Sending connection context
0923 19:35:07.218191 (+ 338us) client_negotiation.cc:240] Negotiation successful
0923 19:35:07.218806 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":44,"thread_start_us":484,"threads_started":1}
I20260923 19:35:07.220544 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.196500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.0.97.60:42947)
0923 19:35:07.197351 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.197363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.197408 (+ 45us) server_negotiation.cc:407] Connection header received
0923 19:35:07.199363 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.199387 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.199827 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.200172 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.202115 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.204272 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.206101 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.207110 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.207309 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.211901 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.211913 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.211930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.211967 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.215768 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.216546 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.216556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.216560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.216720 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.217074 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.217079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.217082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.217853 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.218195 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.218525 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.218931 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":204,"server-negotiator.queue_time_us":635,"thread_start_us":348,"threads_started":1}
I20260923 19:35:07.222553 23005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:07 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 23 19:35:07 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:35:07.237594 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.223300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37947 (local address 127.0.0.1:59384)
0923 19:35:07.223586 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:07.223607 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:07.223724 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:07.224093 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:07.224097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:07.224607 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:07.224825 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.224837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.225966 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.225972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.227169 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.227182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.227970 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.227977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.228216 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.228905 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:07.228928 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:07.231390 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:07.234917 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.234923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.234927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.235453 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.235851 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.235855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.235858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.236026 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.236683 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:07.236688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:07.236878 (+ 190us) client_negotiation.cc:769] Sending connection context
0923 19:35:07.237007 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 19:35:07.237279 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260923 19:35:07.237900 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.223505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.0.97.61:37947)
0923 19:35:07.223705 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.223712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.223736 (+ 24us) server_negotiation.cc:407] Connection header received
0923 19:35:07.223850 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.223856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.223942 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.224058 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.224965 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.225802 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.227307 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.227804 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.227898 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.231584 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.231588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.231592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.231623 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.234755 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.235592 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.235596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.235598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.235740 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.236181 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.236186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.236188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.236543 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.236684 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.237170 (+ 486us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.237422 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 19:35:07.239099 23005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:35:07.240108 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.224498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.0.97.62:35665)
0923 19:35:07.224633 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.224637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.224887 (+ 250us) server_negotiation.cc:407] Connection header received
0923 19:35:07.225093 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.225098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.225181 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.225285 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.226682 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.227380 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.228992 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.229399 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.229493 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.233854 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.233858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.233861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.233893 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.236685 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.237682 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.237689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.237693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.237887 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.238499 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.238503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.238506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.238907 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.239220 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.239514 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.239790 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 23 19:35:07 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 19:35:07 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:35:07.253494 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.239871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:58004)
0923 19:35:07.240168 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:07.240189 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:07.240300 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:07.240644 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:07.240649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:07.241084 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:07.241261 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.241273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.242472 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.242478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.243593 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.243604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.244223 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.244228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.244430 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.245088 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:07.245110 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:07.247344 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:07.250787 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.250793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.250797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.251428 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.251798 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.251803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.251806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.251939 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.252515 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:07.252520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:07.252689 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 19:35:07.252831 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 19:35:07.253100 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260923 19:35:07.253738 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.240002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.0.97.60:42947)
0923 19:35:07.240214 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.240220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.240246 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:35:07.240421 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.240426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.240496 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.240618 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.241395 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.242359 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.243711 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.244097 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.244199 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.247544 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.247551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.247555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.247590 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.250605 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.251562 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.251565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.251568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.251695 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.252040 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.252044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.252046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.252383 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.252542 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.252961 (+ 419us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.253276 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":93}
I20260923 19:35:07.258705 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.242686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.0.97.61:37947)
0923 19:35:07.242877 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.242882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.242902 (+ 20us) server_negotiation.cc:407] Connection header received
0923 19:35:07.242958 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.242963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.243032 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.243139 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.244301 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.245214 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.246479 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.247083 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.247215 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.252449 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.252457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.252462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.252508 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.255819 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.256767 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.256771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.256773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.256930 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.257366 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.257371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.257374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.257659 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.257898 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.258211 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.258429 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 19:35:07.259624 23005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67: Bootstrap starting.
I20260923 19:35:07.263484 23005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67: Neither blocks nor log segments found. Creating new log.
I20260923 19:35:07.264608 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:35:07.264734 23005 log.cc:824] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67: Log is configured to *not* fsync() on all Append() calls
I20260923 19:35:07.268446 23005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67: No bootstrap required, opened a new log
I20260923 19:35:07.267673 22831 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54734
I20260923 19:35:07.276389 23005 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } }
I20260923 19:35:07.276793 23005 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae851d92d0814d70badcb3bb55bd1a67, State: Initialized, Role: FOLLOWER
I20260923 19:35:07.277297 23005 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } }
I20260923 19:35:07.278481 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.279008 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:07.279579 23005 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 19:35:07.289444 23023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:35:07.289609 23023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:35:07.287693 22938 master_runner.cc:186] Error getting master registration for 127.0.97.62:35665: OK, instance_id { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" instance_seqno: 1790192106028852 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 19:35:07.299652 22938 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 19:35:07.346405 22930 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 5
Sep 23 19:35:07 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 19:35:07.361310 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.347220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.0.97.60:42947)
0923 19:35:07.347437 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.347441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.347638 (+ 197us) server_negotiation.cc:407] Connection header received
0923 19:35:07.347869 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.347873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.347953 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.348052 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.348869 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.349645 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.350891 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.351366 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.351477 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.355634 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.355640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.355644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.355679 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.358679 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.359353 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.359357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.359359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.359474 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.359836 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.359841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.359844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.360213 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.360383 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.360736 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.360947 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 19:35:07.361304 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.347179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:58018)
0923 19:35:07.347486 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:07.347518 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:07.347724 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:07.348126 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:07.348132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:07.348561 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:07.348730 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.348739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.349787 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.349793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.350760 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.350773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.351504 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.351509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.351708 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.352435 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:07.352460 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:07.355410 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:07.358824 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.358830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.358834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.359239 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.359580 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.359586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.359590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.359737 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.360326 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:07.360332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:07.360484 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 19:35:07.360607 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 19:35:07.360915 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 19:35:07.365980 22930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd: Bootstrap starting.
I20260923 19:35:07.369987 22930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd: Neither blocks nor log segments found. Creating new log.
I20260923 19:35:07.371310 22930 log.cc:824] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd: Log is configured to *not* fsync() on all Append() calls
I20260923 19:35:07.374006 22930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd: No bootstrap required, opened a new log
I20260923 19:35:07.384007 22930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } }
I20260923 19:35:07.384459 22930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1bc24539a2475baaa7086c9e7c1fcd, State: Initialized, Role: FOLLOWER
I20260923 19:35:07.384967 22930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } }
I20260923 19:35:07.385946 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.386353 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:07.387060 22930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [sys.catalog]: configured and running, proceeding with master startup.
W20260923 19:35:07.396075 23037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:35:07.396277 23037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:35:07.394426 22863 master_runner.cc:186] Error getting master registration for 127.0.97.62:35665: OK, instance_id { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" instance_seqno: 1790192106028852 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 19:35:07.406626 22863 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 19:35:07.415848 22959 tablet.cc:2369] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:35:07.448379 22856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } attempt: 5
Sep 23 19:35:07 dist-test-slave-kxfb krb5kdc[22776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 19:35:07.467588 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.449732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.0.97.60:42947)
0923 19:35:07.449917 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.449924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.449948 (+ 24us) server_negotiation.cc:407] Connection header received
0923 19:35:07.450026 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.450032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.450150 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.450266 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.451379 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.452538 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.454408 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.455097 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.455235 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.459727 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.459734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.459739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.459778 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.463559 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.464734 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.464739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.464743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.464915 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.465426 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.465431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.465435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.465875 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.466022 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.466919 (+ 897us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.467178 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 19:35:07.468408 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.449299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42947 (local address 127.0.0.1:58028)
0923 19:35:07.449602 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:07.449634 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:07.449812 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:07.450379 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:07.450386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:07.450939 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:07.451198 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.451210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.452693 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.452700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.454202 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:07.454218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.455261 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.455267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:07.455481 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.456293 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:07.456323 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:07.459463 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:07.463805 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.463811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.463816 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.464566 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.465057 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:07.465062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:07.465066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:07.465290 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:07.466087 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:07.466092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:07.466283 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 19:35:07.466449 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 19:35:07.468007 (+ 1558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 19:35:07.475845 22856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6: Bootstrap starting.
I20260923 19:35:07.481148 22856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6: Neither blocks nor log segments found. Creating new log.
I20260923 19:35:07.482898 22856 log.cc:824] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6: Log is configured to *not* fsync() on all Append() calls
I20260923 19:35:07.487706 22856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6: No bootstrap required, opened a new log
I20260923 19:35:07.496619 22856 raft_consensus.cc:348] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } }
I20260923 19:35:07.497107 22856 raft_consensus.cc:729] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65817382bd54c9a9ab7cdc3cc0400c6, State: Initialized, Role: FOLLOWER
I20260923 19:35:07.497786 22856 consensus_queue.cc:260] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } }
I20260923 19:35:07.505034 23040 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.505487 23040 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:07.507675 22856 sys_catalog.cc:564] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:35:07.507926 22831 raft_consensus.cc:482] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 19:35:07.508275 22831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:35:07.512008 22831 raft_consensus.cc:504] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } }
I20260923 19:35:07.515043 22980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ae851d92d0814d70badcb3bb55bd1a67"
I20260923 19:35:07.514981 22905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd"
I20260923 19:35:07.515555 22980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:35:07.515666 22905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [term 0 FOLLOWER]: Advancing to term 1
W20260923 19:35:07.518131 22810 tablet.cc:2369] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:35:07.519328 22980 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c65817382bd54c9a9ab7cdc3cc0400c6 in term 1.
I20260923 19:35:07.519578 22905 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c65817382bd54c9a9ab7cdc3cc0400c6 in term 1.
I20260923 19:35:07.520047 22808 leader_election.cc:302] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae851d92d0814d70badcb3bb55bd1a67, c65817382bd54c9a9ab7cdc3cc0400c6; no voters:
I20260923 19:35:07.520548 23040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 19:35:07.520841 22831 leader_election.cc:288] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [CANDIDATE]: Term 1 election: Requested vote from peers 9b1bc24539a2475baaa7086c9e7c1fcd (127.0.97.61:37947), ae851d92d0814d70badcb3bb55bd1a67 (127.0.97.60:42947)
I20260923 19:35:07.534658 23040 raft_consensus.cc:686] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [term 1 LEADER]: Becoming Leader. State: Replica: c65817382bd54c9a9ab7cdc3cc0400c6, State: Running, Role: LEADER
I20260923 19:35:07.535593 23040 consensus_queue.cc:237] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } }
I20260923 19:35:07.550513 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c65817382bd54c9a9ab7cdc3cc0400c6. Latest consensus state: current_term: 1 leader_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.551085 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:07.559109 23051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:35:07.578169 23051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260923 19:35:07.588747 22884 tablet.cc:2369] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:35:07.595815 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.547013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.0.97.62:35665)
0923 19:35:07.547211 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.547217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.554238 (+ 7021us) server_negotiation.cc:407] Connection header received
0923 19:35:07.555549 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.555556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.555646 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.555720 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.559966 (+ 4246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.561237 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.567268 (+ 6031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.568012 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.568115 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.579431 (+ 11316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.579438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.579444 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.579542 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.583753 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.585335 (+ 1582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.585341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.585346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.585499 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.589570 (+ 4071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.589576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.589580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.590056 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.590440 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.595088 (+ 4648us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.595370 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 19:35:07.612545 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.554899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.0.97.61:37947)
0923 19:35:07.555187 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.555195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.564626 (+ 9431us) server_negotiation.cc:407] Connection header received
0923 19:35:07.566047 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.566054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.566146 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.566231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.574893 (+ 8662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.576069 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.587300 (+ 11231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.588050 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.588180 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.602264 (+ 14084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.602273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.602278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.602370 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.606094 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.606996 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.607001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.607004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.607170 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.607621 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.607626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.607629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.611010 (+ 3381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.611271 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.611881 (+ 610us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.612150 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 19:35:07.620920 22980 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [term 1 FOLLOWER]: Refusing update from remote peer c65817382bd54c9a9ab7cdc3cc0400c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:35:07.621996 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:07.565169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.0.97.60:42947)
0923 19:35:07.565383 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.565390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.574480 (+ 9090us) server_negotiation.cc:407] Connection header received
0923 19:35:07.577966 (+ 3486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.577977 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.578095 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.578166 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.585690 (+ 7524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.586888 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.598275 (+ 11387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:07.599088 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:07.599193 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:07.607042 (+ 7849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:07.607050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:07.607055 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:07.607097 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:07.610531 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.619107 (+ 8576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.619113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.619116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.619302 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:07.619811 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:07.619817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:07.619820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:07.620274 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:07.620592 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:07.621247 (+ 655us) server_negotiation.cc:299] Negotiation successful
0923 19:35:07.621558 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 19:35:07.622817 23042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:35:07.631263 22905 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [term 1 FOLLOWER]: Refusing update from remote peer c65817382bd54c9a9ab7cdc3cc0400c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:35:07.632306 23042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:35:07.649174 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c65817382bd54c9a9ab7cdc3cc0400c6. Latest consensus state: current_term: 1 leader_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.649554 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:07.654191 23064 mvcc.cc:205] Tried to move back new op lower bound from 7332626872745148416 to 7332626872473387008. Current Snapshot: MvccSnapshot[applied={T|T < 7332626872745148416}]
I20260923 19:35:07.658126 23061 mvcc.cc:205] Tried to move back new op lower bound from 7332626872745148416 to 7332626872473387008. Current Snapshot: MvccSnapshot[applied={T|T < 7332626872745148416}]
I20260923 19:35:07.669477 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c65817382bd54c9a9ab7cdc3cc0400c6. Latest consensus state: current_term: 1 leader_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.669859 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:07.680779 23040 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.681209 23040 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:07.682717 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.683038 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:07.691753 23051 catalog_manager.cc:1347] Generated new cluster ID: 35ec3d0454cd421b821eb9a7e7395cd9
I20260923 19:35:07.691876 23051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:35:07.708496 23040 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.709048 23040 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6 [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:07.710944 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65817382bd54c9a9ab7cdc3cc0400c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35665 } } peers { permanent_uuid: "9b1bc24539a2475baaa7086c9e7c1fcd" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37947 } } peers { permanent_uuid: "ae851d92d0814d70badcb3bb55bd1a67" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42947 } } }
I20260923 19:35:07.711261 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:07.731952 23051 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:35:07.733790 23051 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:35:07.756910 23051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c65817382bd54c9a9ab7cdc3cc0400c6: Generated new TSK 0
I20260923 19:35:07.758165 23051 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 19:35:07.856824 23010 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.852447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.0.97.62:35665)
0923 19:35:07.852676 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.852682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.854925 (+ 2243us) server_negotiation.cc:407] Connection header received
0923 19:35:07.855215 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.855221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.855305 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.855376 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.856547 (+ 1171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54744: BlockingRecv error: recv got EOF from 127.0.0.1:54744 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 19:35:07.859292 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.855237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.97.61:37947)
0923 19:35:07.855487 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.855495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.857594 (+ 2099us) server_negotiation.cc:407] Connection header received
0923 19:35:07.857903 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.857909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.858010 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.858088 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.859049 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59418: BlockingRecv error: recv got EOF from 127.0.0.1:59418 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260923 19:35:07.862504 23008 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.860521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.0.97.60:42947)
0923 19:35:07.860708 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.860714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.861015 (+ 301us) server_negotiation.cc:407] Connection header received
0923 19:35:07.861131 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.861137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.861248 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.861388 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.862259 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58048: BlockingRecv error: recv got EOF from 127.0.0.1:58048 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 19:35:07.875083 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.872336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.0.97.61:37947)
0923 19:35:07.872573 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.872579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.873380 (+ 801us) server_negotiation.cc:407] Connection header received
0923 19:35:07.873642 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.873649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.873810 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.873963 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.874755 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59434: BlockingRecv error: recv got EOF from 127.0.0.1:59434 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260923 19:35:07.887248 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.884491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.0.97.61:37947)
0923 19:35:07.884683 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.884689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.885620 (+ 931us) server_negotiation.cc:407] Connection header received
0923 19:35:07.885848 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.885854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.885967 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.886099 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.886982 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59438: BlockingRecv error: recv got EOF from 127.0.0.1:59438 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 19:35:07.900585 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.896255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.0.97.61:37947)
0923 19:35:07.896445 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.896451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.898895 (+ 2444us) server_negotiation.cc:407] Connection header received
0923 19:35:07.899106 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.899112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.899237 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.899365 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.900274 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59454: BlockingRecv error: recv got EOF from 127.0.0.1:59454 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260923 19:35:07.915504 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.912404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.0.97.61:37947)
0923 19:35:07.912711 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.912719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.913752 (+ 1033us) server_negotiation.cc:407] Connection header received
0923 19:35:07.914022 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.914029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.914140 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.914269 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.915265 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59460: BlockingRecv error: recv got EOF from 127.0.0.1:59460 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
W20260923 19:35:07.931226 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.928655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.0.97.61:37947)
0923 19:35:07.928917 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.928923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.929501 (+ 578us) server_negotiation.cc:407] Connection header received
0923 19:35:07.929775 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.929781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.929915 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.930036 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.930954 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59476: BlockingRecv error: recv got EOF from 127.0.0.1:59476 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260923 19:35:07.954058 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.951871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.0.97.61:37947)
0923 19:35:07.952098 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.952105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.952456 (+ 351us) server_negotiation.cc:407] Connection header received
0923 19:35:07.952675 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.952681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.952788 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.952942 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.953810 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59486: BlockingRecv error: recv got EOF from 127.0.0.1:59486 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260923 19:35:07.992292 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:07.990174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.0.97.61:37947)
0923 19:35:07.990431 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:07.990438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:07.990787 (+ 349us) server_negotiation.cc:407] Connection header received
0923 19:35:07.990996 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:07.991002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:07.991107 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:07.991269 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:07.992022 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59500: BlockingRecv error: recv got EOF from 127.0.0.1:59500 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260923 19:35:08.063774 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:08.061422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.0.97.61:37947)
0923 19:35:08.061732 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:08.061740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:08.062146 (+ 406us) server_negotiation.cc:407] Connection header received
0923 19:35:08.062382 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:08.062389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:08.062496 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:08.062656 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:08.063524 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59512: BlockingRecv error: recv got EOF from 127.0.0.1:59512 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260923 19:35:08.197649 22935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:08.195626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.0.97.61:37947)
0923 19:35:08.195871 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:08.195878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:08.196105 (+ 227us) server_negotiation.cc:407] Connection header received
0923 19:35:08.196307 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:08.196313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:08.196429 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:08.196582 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:08.197378 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59516: BlockingRecv error: recv got EOF from 127.0.0.1:59516 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 19:35:08.291728 23023 catalog_manager.cc:1259] Loaded cluster ID: 35ec3d0454cd421b821eb9a7e7395cd9
I20260923 19:35:08.291839 23023 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67: loading cluster ID for follower catalog manager: success
I20260923 19:35:08.294265 23023 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67: acquiring CA information for follower catalog manager: success
I20260923 19:35:08.296013 23023 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ae851d92d0814d70badcb3bb55bd1a67: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 19:35:08.398341 23037 catalog_manager.cc:1259] Loaded cluster ID: 35ec3d0454cd421b821eb9a7e7395cd9
I20260923 19:35:08.398439 23037 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd: loading cluster ID for follower catalog manager: success
I20260923 19:35:08.400830 23037 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd: acquiring CA information for follower catalog manager: success
I20260923 19:35:08.402696 23037 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b1bc24539a2475baaa7086c9e7c1fcd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 19:35:08.463284 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:08.457149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.0.97.61:37947)
0923 19:35:08.457458 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:08.457466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:08.457624 (+ 158us) server_negotiation.cc:407] Connection header received
0923 19:35:08.457820 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:08.457839 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:08.458033 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:08.458159 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:08.458658 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.459441 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.460836 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.461242 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.461352 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:08.461870 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:08.462613 (+ 743us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:08.462739 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:08.462863 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 19:35:08.462948 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260923 19:35:08.482129 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:08.474869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.0.97.60:42947)
0923 19:35:08.475482 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:08.475490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:08.475614 (+ 124us) server_negotiation.cc:407] Connection header received
0923 19:35:08.475915 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:08.475920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:08.476112 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:08.476368 (+ 256us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:08.476868 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.477788 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.479404 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.479930 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.480051 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:08.480612 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:08.481360 (+ 748us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:08.481515 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:08.481618 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 19:35:08.481720 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":319,"threads_started":1}
I20260923 19:35:08.496726 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:08.489597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.0.97.61:37947)
0923 19:35:08.489797 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:08.489802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:08.490268 (+ 466us) server_negotiation.cc:407] Connection header received
0923 19:35:08.490513 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:08.490518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:08.490605 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:08.490690 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:08.491533 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.492583 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.494693 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.495350 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.495469 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:08.495951 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:08.496092 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:08.496180 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:08.496332 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 19:35:08.496417 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 19:35:08.496999 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:08.489018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.0.97.62:35665)
0923 19:35:08.489623 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:08.489629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:08.489655 (+ 26us) server_negotiation.cc:407] Connection header received
0923 19:35:08.489839 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:08.489846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:08.490069 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:08.490844 (+ 775us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:08.491015 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.491938 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.494024 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.494691 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.494824 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:08.495352 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:08.495835 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:08.495931 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:08.496612 (+ 681us) server_negotiation.cc:299] Negotiation successful
0923 19:35:08.496700 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":283,"threads_started":1}
I20260923 19:35:08.497272 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:08.490573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.0.97.60:42947)
0923 19:35:08.490782 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:08.490789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:08.491454 (+ 665us) server_negotiation.cc:407] Connection header received
0923 19:35:08.491665 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:08.491671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:08.491740 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:08.491809 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:08.492442 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.493317 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.495391 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:08.495905 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:08.496027 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:08.496493 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:08.496631 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:08.496690 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:08.496895 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 19:35:08.496983 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 19:35:08.502876 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22789
I20260923 19:35:08.504145 22851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:35:08.625275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22789
I20260923 19:35:08.647162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22863
I20260923 19:35:08.648403 22925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:35:08.771013 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22863
I20260923 19:35:08.791725 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 22938
I20260923 19:35:08.792802 23000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:35:08.901219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 22938
2026-09-23T19:35:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4721 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 19:35:09 dist-test-slave-kxfb krb5kdc[23150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 19:35:09 dist-test-slave-kxfb krb5kdc[23150](info): set up 2 sockets
Sep 23 19:35:09 dist-test-slave-kxfb krb5kdc[23150](info): commencing operation
krb5kdc: starting...
W20260923 19:35:09.938714 388 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/test-user.keytab.
Sep 23 19:35:10 dist-test-slave-kxfb krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T19:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T19:35:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 19:35:10.233584 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36317
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36317,127.0.97.61:40553,127.0.97.60:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 19:35:10.586041 23163 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 19:35:10.586432 23163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:35:10.586537 23163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:35:10.586655 23163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:35:10.593950 23163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:35:10.594053 23163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:35:10.594110 23163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:35:10.594178 23163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:35:10.594265 23163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:35:10.604274 23163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36317,127.0.97.61:40553,127.0.97.60:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:35:10.605592 23163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:35:10.606915 23163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:35:10.619673 23168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:10.619678 23171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:10.619691 23169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:35:10.620658 23163 server_base.cc:1034] running on GCE node
I20260923 19:35:10.621618 23163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:35:10.623479 23163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:35:10.624745 23163 hybrid_clock.cc:648] HybridClock initialized: now 1790192110624652 us; error 61 us; skew 500 ppm
I20260923 19:35:10.625167 23163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:35:10 dist-test-slave-kxfb krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:35:10.628777 23163 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 19:35:10.630491 23163 webserver.cc:460] Webserver started at http://127.0.97.62:46057/ using document root <none> and password file <none>
I20260923 19:35:10.631307 23163 fs_manager.cc:359] Metadata directory not provided
I20260923 19:35:10.631409 23163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:35:10.631722 23163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:35:10.635252 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/data/instance:
uuid: "e49fd225907f436c851d219451f16c05"
format_stamp: "Formatted at 2026-09-23 19:35:10 on dist-test-slave-kxfb"
server_key: "0c311097021b53ab35ecafbf3dc6f18e"
server_key_iv: "98eb556e297cee7040d1f5ca1152057d"
server_key_version: "encryptionkey@0"
I20260923 19:35:10.636150 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/wal/instance:
uuid: "e49fd225907f436c851d219451f16c05"
format_stamp: "Formatted at 2026-09-23 19:35:10 on dist-test-slave-kxfb"
server_key: "0c311097021b53ab35ecafbf3dc6f18e"
server_key_iv: "98eb556e297cee7040d1f5ca1152057d"
server_key_version: "encryptionkey@0"
I20260923 19:35:10.642403 23163 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260923 19:35:10.646595 23178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:35:10.648386 23163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 19:35:10.648589 23163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/wal
uuid: "e49fd225907f436c851d219451f16c05"
format_stamp: "Formatted at 2026-09-23 19:35:10 on dist-test-slave-kxfb"
server_key: "0c311097021b53ab35ecafbf3dc6f18e"
server_key_iv: "98eb556e297cee7040d1f5ca1152057d"
server_key_version: "encryptionkey@0"
I20260923 19:35:10.648823 23163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:35:10.667994 23163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:35:10.673512 23163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:35:10.673801 23163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:35:10.694085 23163 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36317
I20260923 19:35:10.694128 23229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36317 every 8 connection(s)
I20260923 19:35:10.695817 23163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/data/info.pb
I20260923 19:35:10.700165 23230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:35:10.703595 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 23163
I20260923 19:35:10.703809 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-0/wal/instance
I20260923 19:35:10.704310 388 external_mini_cluster.cc:1428] Setting key 261b3abd283179811fc6859517ecdba4
I20260923 19:35:10.707319 23230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:10 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:35:10.731034 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.712922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.0.97.62:36317)
0923 19:35:10.713691 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:10.713699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:10.713752 (+ 53us) server_negotiation.cc:407] Connection header received
0923 19:35:10.715000 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:10.715006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:10.715252 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:10.715428 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:10.716737 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:10.717936 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:10.719380 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:10.719919 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:10.720096 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:10.723966 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:10.723974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:10.723988 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:10.724020 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:10.726752 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:10.727414 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:10.727420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:10.727422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:10.727576 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:10.727934 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:10.727937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:10.727939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:10.728433 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:10.728640 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:10.729157 (+ 517us) server_negotiation.cc:299] Negotiation successful
0923 19:35:10.729527 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":551,"thread_start_us":357,"threads_started":1}
I20260923 19:35:10.731061 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.711201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36317 (local address 127.0.0.1:36492)
0923 19:35:10.712300 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:10.712363 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:10.714606 (+ 2243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:10.715478 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:10.715485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:10.716197 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:10.716535 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:10.716553 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:10.718086 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:10.718089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:10.719236 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:10.719253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:10.720161 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:10.720168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:10.720353 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:10.721120 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:10.721148 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:10.723723 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:10.726876 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:10.726888 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:10.726901 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:10.727286 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:10.727678 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:10.727683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:10.727687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:10.727828 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:10.728564 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:10.728572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:10.728780 (+ 208us) client_negotiation.cc:769] Sending connection context
0923 19:35:10.728997 (+ 217us) client_negotiation.cc:240] Negotiation successful
0923 19:35:10.729658 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":192,"thread_start_us":460,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260923 19:35:10.735311 23230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260923 19:35:10.736840 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.736118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45296)
0923 19:35:10.736417 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:10.736637 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260923 19:35:10.737210 23179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
I20260923 19:35:10.738997 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.738465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45312)
0923 19:35:10.738660 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:10.738785 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 19:35:10.739286 23230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40553: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260923 19:35:10.773955 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40553
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36317,127.0.97.61:40553,127.0.97.60:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:35:10.788748 23230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } attempt: 1
I20260923 19:35:10.790194 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.789651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45320)
0923 19:35:10.789905 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:10.790042 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 19:35:10.791389 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.791017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45322)
0923 19:35:10.791165 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:10.791235 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 19:35:10.791587 23230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40553: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
I20260923 19:35:10.859872 23230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } attempt: 2
I20260923 19:35:10.861166 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.860655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45332)
0923 19:35:10.860914 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:10.861009 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 19:35:10.862337 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.861951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45340)
0923 19:35:10.862113 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:10.862178 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 19:35:10.862505 23230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40553: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
I20260923 19:35:10.953783 23230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } attempt: 3
I20260923 19:35:10.955312 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.954765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45346)
0923 19:35:10.954999 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:10.955107 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 19:35:10.956501 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:10.956117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45352)
0923 19:35:10.956279 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:10.956351 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 19:35:10.956701 23230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40553: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
W20260923 19:35:11.095729 23237 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 19:35:11.096077 23237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:35:11.096189 23237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:35:11.096306 23237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 19:35:11.099992 23230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } attempt: 4
I20260923 19:35:11.101438 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.100849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45366)
0923 19:35:11.101177 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.101288 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260923 19:35:11.102643 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.102245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45372)
0923 19:35:11.102435 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.102514 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260923 19:35:11.102818 23230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40553: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40553: connect: Connection refused (error 111)
W20260923 19:35:11.104192 23237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:35:11.104305 23237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:35:11.104395 23237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:35:11.104465 23237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:35:11.104549 23237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:35:11.114643 23237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36317,127.0.97.61:40553,127.0.97.60:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:35:11.115973 23237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:35:11.117242 23237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:35:11.129065 23243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:11.129096 23245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:11.129187 23242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:35:11.136267 23237 server_base.cc:1034] running on GCE node
I20260923 19:35:11.137264 23237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:35:11.138967 23237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:35:11.140229 23237 hybrid_clock.cc:648] HybridClock initialized: now 1790192111140124 us; error 84 us; skew 500 ppm
I20260923 19:35:11.140584 23237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:35:11.144269 23237 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260923 19:35:11.146126 23237 webserver.cc:460] Webserver started at http://127.0.97.61:38043/ using document root <none> and password file <none>
I20260923 19:35:11.146953 23237 fs_manager.cc:359] Metadata directory not provided
I20260923 19:35:11.147061 23237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:35:11.147394 23237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:35:11.150635 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/data/instance:
uuid: "2980ee3690b44fb69ce5a0f1fe370487"
format_stamp: "Formatted at 2026-09-23 19:35:11 on dist-test-slave-kxfb"
server_key: "6179568173d7d4d299a7ed3afeeb7dd9"
server_key_iv: "17cdea4fab5d26a340fbb64b454f5c68"
server_key_version: "encryptionkey@0"
I20260923 19:35:11.151506 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/wal/instance:
uuid: "2980ee3690b44fb69ce5a0f1fe370487"
format_stamp: "Formatted at 2026-09-23 19:35:11 on dist-test-slave-kxfb"
server_key: "6179568173d7d4d299a7ed3afeeb7dd9"
server_key_iv: "17cdea4fab5d26a340fbb64b454f5c68"
server_key_version: "encryptionkey@0"
I20260923 19:35:11.157455 23237 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 19:35:11.161464 23252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:35:11.163242 23237 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 19:35:11.163434 23237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/wal
uuid: "2980ee3690b44fb69ce5a0f1fe370487"
format_stamp: "Formatted at 2026-09-23 19:35:11 on dist-test-slave-kxfb"
server_key: "6179568173d7d4d299a7ed3afeeb7dd9"
server_key_iv: "17cdea4fab5d26a340fbb64b454f5c68"
server_key_version: "encryptionkey@0"
I20260923 19:35:11.163647 23237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:35:11.185284 23237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:35:11.189747 23237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:35:11.190021 23237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:35:11.209674 23237 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40553
I20260923 19:35:11.209717 23303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40553 every 8 connection(s)
I20260923 19:35:11.211148 23237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/data/info.pb
I20260923 19:35:11.213086 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 23237
I20260923 19:35:11.213311 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-1/wal/instance
I20260923 19:35:11.213959 388 external_mini_cluster.cc:1428] Setting key 4b537cab59fdfef8b38dc710d4c157f3
I20260923 19:35:11.217176 23304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:35:11.227525 23304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260923 19:35:11.252486 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.231226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.0.97.62:36317)
0923 19:35:11.231425 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.231430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.233153 (+ 1723us) server_negotiation.cc:407] Connection header received
0923 19:35:11.235486 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.235492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.235597 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.235694 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.237923 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.238774 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.240628 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.241167 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.241292 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.245417 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.245423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.245427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.245467 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.248854 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.249961 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.249967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.249971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.250143 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.250592 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.250597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.250599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.250991 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.251139 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.251867 (+ 728us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.252123 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 19:35:11.253412 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.231975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36317 (local address 127.0.0.1:36500)
0923 19:35:11.232956 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.233004 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:11.235140 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:11.236005 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:11.236015 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:11.236687 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:11.237680 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.237707 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.238966 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.238970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.240457 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.240481 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.241438 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.241446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.241728 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.242551 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:11.242577 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:11.245103 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:11.249132 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.249142 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.249161 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.249805 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.250249 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.250252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.250254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.250488 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.251106 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:11.251112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:11.251485 (+ 373us) client_negotiation.cc:769] Sending connection context
0923 19:35:11.251725 (+ 240us) client_negotiation.cc:240] Negotiation successful
0923 19:35:11.252182 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":404,"threads_started":1}
I20260923 19:35:11.255527 23304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:35:11.271637 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.256519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45378)
0923 19:35:11.256704 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.256724 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:11.256820 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:11.257929 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:11.257933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:11.258413 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:11.258561 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.258571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.259938 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.259942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.260779 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.260790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.261590 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.261596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.261767 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.262336 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:11.262354 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:11.264940 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:11.268518 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.268525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.268530 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.269154 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.269588 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.269592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.269595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.269796 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.270490 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:11.270495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:11.270688 (+ 193us) client_negotiation.cc:769] Sending connection context
0923 19:35:11.270840 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 19:35:11.271193 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 19:35:11.271688 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.256811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.0.97.61:40553)
0923 19:35:11.257526 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.257532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.257559 (+ 27us) server_negotiation.cc:407] Connection header received
0923 19:35:11.257603 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.257609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.257788 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.257920 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.258723 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.259790 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.260905 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.261447 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.261554 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.265239 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.265253 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.265277 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.265316 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.268307 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.269328 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.269334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.269337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.269474 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.269916 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.269919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.269921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.270356 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.270533 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.270978 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.271325 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":564,"thread_start_us":409,"threads_started":1}
I20260923 19:35:11.274317 23304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:35:11.275728 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.275169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45078)
0923 19:35:11.275385 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.275533 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260923 19:35:11.276000 23256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
I20260923 19:35:11.277734 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.277366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45086)
0923 19:35:11.277519 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.277605 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 19:35:11.277976 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36395: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260923 19:35:11.286403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
/tmp/dist-test-taskhSGBcM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36395
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36317,127.0.97.61:40553,127.0.97.60:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 19:35:11.327399 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } attempt: 1
I20260923 19:35:11.328994 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.328337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45092)
0923 19:35:11.328615 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.328763 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 19:35:11.330201 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.329777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45108)
0923 19:35:11.329953 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.330056 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 19:35:11.330499 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36395: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
I20260923 19:35:11.398800 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } attempt: 2
I20260923 19:35:11.400552 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.399746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45124)
0923 19:35:11.400106 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.400280 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260923 19:35:11.401886 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.401423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45140)
0923 19:35:11.401606 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.401716 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 19:35:11.402222 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36395: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
I20260923 19:35:11.402114 23230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } attempt: 5
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:35:11.416095 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.402892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.0.97.61:40553)
0923 19:35:11.403096 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.403102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.403121 (+ 19us) server_negotiation.cc:407] Connection header received
0923 19:35:11.403299 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.403305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.403368 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.403446 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.404252 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.404949 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.406082 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.406576 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.406694 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.410399 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.410404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.410407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.410441 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.413229 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.413921 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.413927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.413929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.414048 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.414428 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.414432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.414434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.414738 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.414898 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.415634 (+ 736us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.415819 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 19:35:11.416445 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.402812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45390)
0923 19:35:11.403009 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.403031 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:11.403160 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:11.403488 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:11.403493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:11.403914 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:11.404108 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.404118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.405086 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.405091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.405932 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.405944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.406731 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.406737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.406951 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.407673 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:11.407691 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:11.410157 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:11.413410 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.413416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.413419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.413806 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.414152 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.414156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.414160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.414319 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.414974 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:11.414982 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:11.415257 (+ 275us) client_negotiation.cc:769] Sending connection context
0923 19:35:11.415493 (+ 236us) client_negotiation.cc:240] Negotiation successful
0923 19:35:11.415885 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 19:35:11.417541 23230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
I20260923 19:35:11.419085 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.418539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45156)
0923 19:35:11.418752 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.418867 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 19:35:11.420221 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.419850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45160)
0923 19:35:11.420044 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.420123 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 19:35:11.420462 23230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36395: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
I20260923 19:35:11.471688 23230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } attempt: 1
I20260923 19:35:11.472798 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.472368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45176)
0923 19:35:11.472552 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.472636 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 19:35:11.473636 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.473355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45186)
0923 19:35:11.473482 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.473544 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 19:35:11.473816 23230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36395: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
I20260923 19:35:11.493510 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } attempt: 3
I20260923 19:35:11.494990 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.494439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45196)
0923 19:35:11.494714 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.494827 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 19:35:11.496310 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.495865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45212)
0923 19:35:11.496105 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.496182 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260923 19:35:11.496516 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36395: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
I20260923 19:35:11.542068 23230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } attempt: 2
I20260923 19:35:11.543678 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.543082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45214)
0923 19:35:11.543392 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.543508 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260923 19:35:11.544750 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.544416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45218)
0923 19:35:11.544586 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.544652 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 19:35:11.544952 23230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36395: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
I20260923 19:35:11.639748 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } attempt: 4
I20260923 19:35:11.641404 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.640662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45228)
0923 19:35:11.641071 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.641205 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 19:35:11.642509 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.642178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45230)
0923 19:35:11.642348 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.642413 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 19:35:11.642712 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36395: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
I20260923 19:35:11.651177 23230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } attempt: 3
I20260923 19:35:11.652441 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.651958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45244)
0923 19:35:11.652191 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.652294 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 19:35:11.653509 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.653148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45252)
0923 19:35:11.653334 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.653403 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260923 19:35:11.653693 23230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36395: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36395: connect: Connection refused (error 111)
W20260923 19:35:11.680058 23311 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 19:35:11.680547 23311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 19:35:11.680651 23311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 19:35:11.680778 23311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 19:35:11.688800 23311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 19:35:11.688942 23311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 19:35:11.689031 23311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 19:35:11.689102 23311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 19:35:11.689195 23311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 19:35:11.700372 23311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36317,127.0.97.61:40553,127.0.97.60:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 18:36:48 UTC on ccycloud.quasar-figybr.root.comops.site
build id 4668088
ASAN enabled
I20260923 19:35:11.702049 23311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 19:35:11.703684 23311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 19:35:11.717224 23316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:11.717281 23317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 19:35:11.717298 23319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 19:35:11.718256 23311 server_base.cc:1034] running on GCE node
I20260923 19:35:11.719125 23311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 19:35:11.720692 23311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 19:35:11.721906 23311 hybrid_clock.cc:648] HybridClock initialized: now 1790192111721825 us; error 66 us; skew 500 ppm
I20260923 19:35:11.722257 23311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 19:35:11.725545 23311 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260923 19:35:11.727188 23311 webserver.cc:460] Webserver started at http://127.0.97.60:35277/ using document root <none> and password file <none>
I20260923 19:35:11.727916 23311 fs_manager.cc:359] Metadata directory not provided
I20260923 19:35:11.728003 23311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 19:35:11.728288 23311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 19:35:11.731283 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/data/instance:
uuid: "3659d385e6524141b147735b66db15ac"
format_stamp: "Formatted at 2026-09-23 19:35:11 on dist-test-slave-kxfb"
server_key: "eb1bffe2d4e19d5f7e02ec884033b43f"
server_key_iv: "970a428cb9b8ae8917eedf6b82b5d43e"
server_key_version: "encryptionkey@0"
I20260923 19:35:11.731995 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/wal/instance:
uuid: "3659d385e6524141b147735b66db15ac"
format_stamp: "Formatted at 2026-09-23 19:35:11 on dist-test-slave-kxfb"
server_key: "eb1bffe2d4e19d5f7e02ec884033b43f"
server_key_iv: "970a428cb9b8ae8917eedf6b82b5d43e"
server_key_version: "encryptionkey@0"
I20260923 19:35:11.737358 23311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 19:35:11.741011 23326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 19:35:11.742762 23311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 19:35:11.742956 23311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/wal
uuid: "3659d385e6524141b147735b66db15ac"
format_stamp: "Formatted at 2026-09-23 19:35:11 on dist-test-slave-kxfb"
server_key: "eb1bffe2d4e19d5f7e02ec884033b43f"
server_key_iv: "970a428cb9b8ae8917eedf6b82b5d43e"
server_key_version: "encryptionkey@0"
I20260923 19:35:11.743201 23311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 19:35:11.762986 23311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 19:35:11.766906 23311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 19:35:11.767185 23311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 19:35:11.785442 23311 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36395
I20260923 19:35:11.785503 23377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36395 every 8 connection(s)
I20260923 19:35:11.786864 23311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/data/info.pb
I20260923 19:35:11.787237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu as pid 23311
I20260923 19:35:11.787422 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSGBcM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790191744267121-388-0/minicluster-data/master-2/wal/instance
I20260923 19:35:11.787998 388 external_mini_cluster.cc:1428] Setting key c131d5c8fecbb7755428c6a26a199e15
I20260923 19:35:11.792049 23378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 19:35:11.799887 23378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 19:35:11.811498 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.790142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.0.97.60:36395)
0923 19:35:11.791052 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.791066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.791109 (+ 43us) server_negotiation.cc:407] Connection header received
0923 19:35:11.792816 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.792840 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.793271 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.793631 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.795061 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.796427 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.797637 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.798349 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.798522 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.802843 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.802857 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.802881 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.802938 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.806511 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.807163 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.807171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.807179 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.807430 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.807760 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.807765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.807769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.808487 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.808715 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.809329 (+ 614us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.809821 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":377,"threads_started":1}
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 19:35:11.819028 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.802953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36317 (local address 127.0.0.1:36512)
0923 19:35:11.803917 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.803970 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:11.804145 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:11.804541 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:11.804553 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:11.805356 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:11.805644 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.805655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.806757 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.806761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.808287 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.808300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.809266 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.809274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.809459 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.810056 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:11.810077 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:11.812740 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:11.816163 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.816176 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.816183 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.816659 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.817062 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.817065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.817068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.817228 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.817851 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:11.817861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:11.818095 (+ 234us) client_negotiation.cc:769] Sending connection context
0923 19:35:11.818341 (+ 246us) client_negotiation.cc:240] Negotiation successful
0923 19:35:11.818692 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":512,"threads_started":1}
I20260923 19:35:11.819105 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.802920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.0.97.62:36317)
0923 19:35:11.803412 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.803418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.804093 (+ 675us) server_negotiation.cc:407] Connection header received
0923 19:35:11.804305 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.804310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.804372 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.804460 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.805789 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.806593 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.808439 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.809042 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.809157 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.812935 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.812940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.812944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.812977 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.816000 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.816793 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.816800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.816803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.816952 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.817407 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.817410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.817412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.817725 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.817858 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.818482 (+ 624us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.818696 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":229,"threads_started":1}
I20260923 19:35:11.820935 23378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:35:11.830917 23230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } attempt: 4
I20260923 19:35:11.831293 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.814979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.0.97.62:36317)
0923 19:35:11.815689 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.815695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.815722 (+ 27us) server_negotiation.cc:407] Connection header received
0923 19:35:11.815802 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.815808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.815897 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.816012 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.817252 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.818487 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.819959 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.820605 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.820732 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.825494 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.825500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.825503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.825538 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.828650 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.829290 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.829295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.829297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.829435 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.829766 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.829770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.829772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.830055 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.830369 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.830595 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.830841 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":529,"thread_start_us":302,"threads_started":1}
I20260923 19:35:11.838510 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.822077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40553 (local address 127.0.0.1:45400)
0923 19:35:11.822336 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.822358 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:11.822471 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:11.822804 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:11.822809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:11.823280 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:11.823478 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.823490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.824888 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.824894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.825964 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.825975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.826895 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.826899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.827036 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.827525 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:11.827542 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:11.829765 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:11.835131 (+ 5366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.835137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.835142 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.835607 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.836065 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.836070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.836074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.836270 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.837534 (+ 1264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:11.837539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:11.837708 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 19:35:11.837845 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 19:35:11.838131 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 19:35:11.838665 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.822252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.0.97.61:40553)
0923 19:35:11.822463 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.822468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.822485 (+ 17us) server_negotiation.cc:407] Connection header received
0923 19:35:11.822593 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.822598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.822666 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.822747 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.823691 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.824721 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.826113 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.826734 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.826882 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.829992 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.829998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.830002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.830044 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.834942 (+ 4898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.835745 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.835750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.835753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.835922 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.836401 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.836408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.836411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.836867 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.837047 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.837983 (+ 936us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.838276 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260923 19:35:11.839594 23378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 19:35:11.848295 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.831827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.0.97.60:36395)
0923 19:35:11.832058 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.832066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.832090 (+ 24us) server_negotiation.cc:407] Connection header received
0923 19:35:11.832156 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.832162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.832266 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.832387 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.833116 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.834325 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.836212 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.836889 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.837014 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.841145 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.841160 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.841165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.841202 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.844893 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.845877 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.845883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.845886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.846060 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.846666 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.846670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.846673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.846984 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.847096 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.847631 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.847923 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 19:35:11.848387 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.831665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45278)
0923 19:35:11.831859 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.831880 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:11.831981 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:11.832424 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:11.832429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:11.832814 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:11.832973 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.832985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.834522 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.834530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.835975 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.835993 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.837058 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.837065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.837295 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.837839 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:11.837870 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:11.840936 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:11.845080 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.845088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.845093 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.845709 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.846250 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.846256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.846260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.846506 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.847134 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:11.847140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:11.847344 (+ 204us) client_negotiation.cc:769] Sending connection context
0923 19:35:11.847539 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 19:35:11.847922 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 19:35:11.854313 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.833566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.0.97.61:40553)
0923 19:35:11.835613 (+ 2047us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.835620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.835649 (+ 29us) server_negotiation.cc:407] Connection header received
0923 19:35:11.835718 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.835725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.835817 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.835958 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.837191 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.838349 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.840589 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.841311 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.841438 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.846823 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.846832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.846837 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.846887 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.851522 (+ 4635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.852245 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.852249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.852252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.852400 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.852750 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.852755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.852759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.853100 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.853428 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.853681 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.853899 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1898,"thread_start_us":1638,"threads_started":1}
I20260923 19:35:11.856297 23230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05: Bootstrap starting.
I20260923 19:35:11.858160 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.840231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45286)
0923 19:35:11.840542 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.840560 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:11.840656 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:11.842414 (+ 1758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:11.842424 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:11.842972 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:11.843175 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.843190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.844595 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.844601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.845894 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.845909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.847761 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.847768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.848010 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.848830 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:11.848856 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:11.851583 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:11.855826 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.855830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.855833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.856234 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.856588 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.856593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.856597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.856720 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.857199 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:11.857203 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:11.857330 (+ 127us) client_negotiation.cc:769] Sending connection context
0923 19:35:11.857468 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 19:35:11.857812 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260923 19:35:11.859093 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.840229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.0.97.60:36395)
0923 19:35:11.841823 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.841830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.841857 (+ 27us) server_negotiation.cc:407] Connection header received
0923 19:35:11.841914 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.841921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.842025 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.842162 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.843391 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.844452 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.846021 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.846546 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.846676 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.851831 (+ 5155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.851838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.851844 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.851886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.855687 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.856357 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.856362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.856365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.856498 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.856810 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.856815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.856818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.857090 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.857211 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.857633 (+ 422us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.858007 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":1423,"thread_start_us":1077,"threads_started":1}
I20260923 19:35:11.860306 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 19:35:11.861193 23230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05: Neither blocks nor log segments found. Creating new log.
I20260923 19:35:11.862418 23230 log.cc:824] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05: Log is configured to *not* fsync() on all Append() calls
I20260923 19:35:11.864632 23378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac: Bootstrap starting.
I20260923 19:35:11.865790 23230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05: No bootstrap required, opened a new log
I20260923 19:35:11.863351 23205 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e49fd225907f436c851d219451f16c05"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36518
I20260923 19:35:11.868542 23378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac: Neither blocks nor log segments found. Creating new log.
I20260923 19:35:11.869778 23378 log.cc:824] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac: Log is configured to *not* fsync() on all Append() calls
I20260923 19:35:11.872255 23230 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } }
I20260923 19:35:11.872648 23230 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49fd225907f436c851d219451f16c05, State: Initialized, Role: FOLLOWER
I20260923 19:35:11.873124 23230 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } }
I20260923 19:35:11.873343 23378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac: No bootstrap required, opened a new log
I20260923 19:35:11.874140 23389 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:11.874629 23389 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:11.875530 23230 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:35:11.875752 23205 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 19:35:11.876034 23205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:35:11.880092 23205 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } }
I20260923 19:35:11.883373 23205 leader_election.cc:288] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [CANDIDATE]: Term 1 election: Requested vote from peers 2980ee3690b44fb69ce5a0f1fe370487 (127.0.97.61:40553), 3659d385e6524141b147735b66db15ac (127.0.97.60:36395)
I20260923 19:35:11.883366 23378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } }
I20260923 19:35:11.883791 23378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3659d385e6524141b147735b66db15ac, State: Initialized, Role: FOLLOWER
I20260923 19:35:11.884301 23378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } }
I20260923 19:35:11.888195 23163 master_runner.cc:186] Error getting master registration for 127.0.97.61:40553: OK, instance_id { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" instance_seqno: 1790192111204019 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 19:35:11.902194 23163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 19:35:11.906319 23400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 19:35:11.906492 23400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 19:35:11.912029 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:11.912483 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:11.913398 23378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:35:11.913566 23353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e49fd225907f436c851d219451f16c05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3659d385e6524141b147735b66db15ac"
I20260923 19:35:11.914341 23353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:35:11.918371 23353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49fd225907f436c851d219451f16c05 in term 1.
I20260923 19:35:11.919495 23182 leader_election.cc:302] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3659d385e6524141b147735b66db15ac, e49fd225907f436c851d219451f16c05; no voters:
I20260923 19:35:11.920120 23389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [term 1 FOLLOWER]: Leader election won for term 1
W20260923 19:35:11.927320 23184 tablet.cc:2369] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:35:11.930773 23389 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [term 1 LEADER]: Becoming Leader. State: Replica: e49fd225907f436c851d219451f16c05, State: Running, Role: LEADER
I20260923 19:35:11.932174 23389 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } }
I20260923 19:35:11.941901 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } attempt: 5
Sep 23 19:35:11 dist-test-slave-kxfb krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790192111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 19:35:11.961730 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.942771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36395 (local address 127.0.0.1:45306)
0923 19:35:11.943256 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 19:35:11.943277 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 19:35:11.943424 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 19:35:11.944141 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 19:35:11.944145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 19:35:11.944701 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0923 19:35:11.944934 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.944946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.946580 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.946586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.948179 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 19:35:11.948196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.949551 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.949558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 19:35:11.949774 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.950489 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 19:35:11.950522 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 19:35:11.953911 (+ 3389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 19:35:11.958517 (+ 4606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.958524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.958529 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.959184 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.959625 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 19:35:11.959631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 19:35:11.959635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 19:35:11.959814 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 19:35:11.960593 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 19:35:11.960598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 19:35:11.960778 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 19:35:11.960935 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 19:35:11.961289 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349}
I20260923 19:35:11.962816 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.943669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.0.97.60:36395)
0923 19:35:11.943839 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.943845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.943862 (+ 17us) server_negotiation.cc:407] Connection header received
0923 19:35:11.943924 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.943929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.944006 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.944111 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.945113 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.946401 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.948384 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.949115 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.949231 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.954191 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.954200 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.954205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.954256 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.958268 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.959343 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.959349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.959354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.959503 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.959937 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.959942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.959946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.960370 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.960526 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.962181 (+ 1655us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.962439 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 19:35:11.965428 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.903398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.0.97.62:36317)
0923 19:35:11.903671 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.903678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.911449 (+ 7771us) server_negotiation.cc:407] Connection header received
0923 19:35:11.912380 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.912387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.912488 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.912591 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.919806 (+ 7215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.920995 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.928599 (+ 7604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.929432 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.929558 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.945824 (+ 16266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.945835 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.945841 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.945908 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.954432 (+ 8524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.958591 (+ 4159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.958599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.958604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.958804 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.959300 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.959318 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.959322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.959756 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.960064 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.964637 (+ 4573us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.964981 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 19:35:11.969448 23304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487: Bootstrap starting.
I20260923 19:35:11.974176 23304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487: Neither blocks nor log segments found. Creating new log.
I20260923 19:35:11.986621 23304 log.cc:824] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487: Log is configured to *not* fsync() on all Append() calls
I20260923 19:35:11.986416 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49fd225907f436c851d219451f16c05. Latest consensus state: current_term: 1 leader_uuid: "e49fd225907f436c851d219451f16c05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:11.987118 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.904497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.0.97.60:36395)
0923 19:35:11.904711 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.904717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.925654 (+ 20937us) server_negotiation.cc:407] Connection header received
0923 19:35:11.926203 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.926210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.926346 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.926426 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.939457 (+ 13031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.940785 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.956460 (+ 15675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.957267 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.957407 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.971196 (+ 13789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.971207 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.971213 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.971273 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.975484 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.982238 (+ 6754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.982245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.982249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.982498 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.984375 (+ 1877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.984383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.984386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.984930 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.985260 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.986241 (+ 981us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.986603 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 19:35:11.987237 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:11.996515 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:11.902145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.0.97.61:40553)
0923 19:35:11.902432 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:11.902440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:11.933339 (+ 30899us) server_negotiation.cc:407] Connection header received
0923 19:35:11.934047 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:11.934054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:11.934155 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:11.934234 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:11.942389 (+ 8155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.943603 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.962301 (+ 18698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:11.963077 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:11.963189 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:11.977054 (+ 13865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 19:35:11.977063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 19:35:11.977069 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 19:35:11.977128 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 19:35:11.981065 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.987853 (+ 6788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.987861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.987866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.988084 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 19:35:11.991816 (+ 3732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 19:35:11.991825 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 19:35:11.991830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 19:35:11.992356 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 19:35:11.992691 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:11.995727 (+ 3036us) server_negotiation.cc:299] Negotiation successful
0923 19:35:11.996083 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 19:35:12.001479 23421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 19:35:12.003573 23304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487: No bootstrap required, opened a new log
I20260923 19:35:12.007407 23421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 19:35:12.008409 23424 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260923 19:35:12.019644 23332 tablet.cc:2369] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 19:35:12.023119 23353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [term 1 FOLLOWER]: Refusing update from remote peer e49fd225907f436c851d219451f16c05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:35:12.024231 23411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:35:12.015384 23311 master_runner.cc:186] Error getting master registration for 127.0.97.61:40553: OK, instance_id { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" instance_seqno: 1790192111204019 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 19:35:12.026338 23311 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 19:35:12.029495 23304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } }
I20260923 19:35:12.029915 23304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2980ee3690b44fb69ce5a0f1fe370487, State: Initialized, Role: FOLLOWER
I20260923 19:35:12.030531 23304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } }
I20260923 19:35:12.041500 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49fd225907f436c851d219451f16c05. Latest consensus state: current_term: 1 leader_uuid: "e49fd225907f436c851d219451f16c05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:12.041841 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:12.043097 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:12.043658 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:12.044466 23304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 19:35:12.044507 23279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e49fd225907f436c851d219451f16c05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2980ee3690b44fb69ce5a0f1fe370487"
I20260923 19:35:12.044988 23279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [term 0 FOLLOWER]: Advancing to term 1
I20260923 19:35:12.051537 23279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49fd225907f436c851d219451f16c05 in term 1.
I20260923 19:35:12.052783 23278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [term 1 FOLLOWER]: Refusing update from remote peer e49fd225907f436c851d219451f16c05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 19:35:12.054611 23425 mvcc.cc:205] Tried to move back new op lower bound from 7332626890832449536 to 7332626890485923840. Current Snapshot: MvccSnapshot[applied={T|T < 7332626890832449536}]
I20260923 19:35:12.055461 23389 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 19:35:12.056660 23426 mvcc.cc:205] Tried to move back new op lower bound from 7332626890832449536 to 7332626890485923840. Current Snapshot: MvccSnapshot[applied={T|T < 7332626890832449536}]
I20260923 19:35:12.063546 23389 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e49fd225907f436c851d219451f16c05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:12.063980 23389 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:12.068007 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e49fd225907f436c851d219451f16c05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:12.068365 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:12.072067 23421 catalog_manager.cc:1347] Generated new cluster ID: 16cd60c3d19446739734e5a74158ffd2
I20260923 19:35:12.072234 23421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 19:35:12.088562 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49fd225907f436c851d219451f16c05. Latest consensus state: current_term: 1 leader_uuid: "e49fd225907f436c851d219451f16c05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:12.088938 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:12.090417 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e49fd225907f436c851d219451f16c05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:12.090896 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 19:35:12.101456 23389 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e49fd225907f436c851d219451f16c05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49fd225907f436c851d219451f16c05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } peers { permanent_uuid: "2980ee3690b44fb69ce5a0f1fe370487" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40553 } } peers { permanent_uuid: "3659d385e6524141b147735b66db15ac" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36395 } } }
I20260923 19:35:12.101861 23389 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05 [sys.catalog]: This master's current role is: LEADER
I20260923 19:35:12.125434 23421 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 19:35:12.127612 23445 catalog_manager.cc:1259] Loaded cluster ID: 16cd60c3d19446739734e5a74158ffd2
I20260923 19:35:12.127769 23445 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487: loading cluster ID for follower catalog manager: success
I20260923 19:35:12.131197 23421 catalog_manager.cc:1504] Loading token signing keys...
I20260923 19:35:12.131318 23445 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487: acquiring CA information for follower catalog manager: success
W20260923 19:35:12.132587 23445 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260923 19:35:12.148092 23421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e49fd225907f436c851d219451f16c05: Generated new TSK 0
I20260923 19:35:12.149369 23421 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 19:35:12.257752 23384 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.243875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.0.97.62:36317)
0923 19:35:12.244092 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.244099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.255031 (+ 10932us) server_negotiation.cc:407] Connection header received
0923 19:35:12.255822 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.255829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.255947 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.256026 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.257409 (+ 1383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36538: BlockingRecv error: recv got EOF from 127.0.0.1:36538 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260923 19:35:12.258227 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.243901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45420 (local address 127.0.97.61:40553)
0923 19:35:12.244146 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.244153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.255975 (+ 11822us) server_negotiation.cc:407] Connection header received
0923 19:35:12.256416 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.256423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.256520 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.256597 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.257944 (+ 1347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45420: BlockingRecv error: recv got EOF from 127.0.0.1:45420 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260923 19:35:12.261090 23386 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.250030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.0.97.60:36395)
0923 19:35:12.250226 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.250232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.259244 (+ 9012us) server_negotiation.cc:407] Connection header received
0923 19:35:12.259601 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.259608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.259701 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.259816 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.260834 (+ 1018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45318: BlockingRecv error: recv got EOF from 127.0.0.1:45318 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 19:35:12.267647 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.265338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.0.97.61:40553)
0923 19:35:12.265543 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.265549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.266106 (+ 557us) server_negotiation.cc:407] Connection header received
0923 19:35:12.266349 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.266355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.266495 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.266605 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.267428 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45432: BlockingRecv error: recv got EOF from 127.0.0.1:45432 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260923 19:35:12.274874 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.272537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.0.97.61:40553)
0923 19:35:12.272756 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.272762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.273319 (+ 557us) server_negotiation.cc:407] Connection header received
0923 19:35:12.273518 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.273527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.273617 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.273725 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.274629 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45448: BlockingRecv error: recv got EOF from 127.0.0.1:45448 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 19:35:12.286538 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.284019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.0.97.61:40553)
0923 19:35:12.284250 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.284257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.284890 (+ 633us) server_negotiation.cc:407] Connection header received
0923 19:35:12.285094 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.285101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.285208 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.285334 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.286247 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45454: BlockingRecv error: recv got EOF from 127.0.0.1:45454 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260923 19:35:12.298529 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.295931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.0.97.61:40553)
0923 19:35:12.296198 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.296204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.296809 (+ 605us) server_negotiation.cc:407] Connection header received
0923 19:35:12.297053 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.297060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.297178 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.297304 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.298229 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45458: BlockingRecv error: recv got EOF from 127.0.0.1:45458 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260923 19:35:12.315521 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.312765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.0.97.61:40553)
0923 19:35:12.312967 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.312972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.313581 (+ 609us) server_negotiation.cc:407] Connection header received
0923 19:35:12.313883 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.313890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.314038 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.314185 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.315201 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45470: BlockingRecv error: recv got EOF from 127.0.0.1:45470 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 19:35:12.338708 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.336002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.0.97.61:40553)
0923 19:35:12.336216 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.336221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.336861 (+ 640us) server_negotiation.cc:407] Connection header received
0923 19:35:12.337066 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.337074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.337218 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.337390 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.338272 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45482: BlockingRecv error: recv got EOF from 127.0.0.1:45482 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260923 19:35:12.377146 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.374404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.0.97.61:40553)
0923 19:35:12.374611 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.374618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.375284 (+ 666us) server_negotiation.cc:407] Connection header received
0923 19:35:12.375533 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.375541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.375681 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.375881 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.376829 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45494: BlockingRecv error: recv got EOF from 127.0.0.1:45494 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 19:35:12.449051 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.446235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.0.97.61:40553)
0923 19:35:12.446483 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.446489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.447112 (+ 623us) server_negotiation.cc:407] Connection header received
0923 19:35:12.447361 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.447369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.447536 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.447673 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.448752 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45496: BlockingRecv error: recv got EOF from 127.0.0.1:45496 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260923 19:35:12.582847 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.580857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.0.97.61:40553)
0923 19:35:12.581156 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.581163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.581460 (+ 297us) server_negotiation.cc:407] Connection header received
0923 19:35:12.581674 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.581680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.581767 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.581853 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.582616 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45504: BlockingRecv error: recv got EOF from 127.0.0.1:45504 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260923 19:35:12.844539 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 19:35:12.842578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.0.97.61:40553)
0923 19:35:12.842759 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:12.842765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:12.843159 (+ 394us) server_negotiation.cc:407] Connection header received
0923 19:35:12.843358 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:12.843363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:12.843436 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:12.843527 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 19:35:12.844290 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45516: BlockingRecv error: recv got EOF from 127.0.0.1:45516 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 19:35:13.013152 23424 catalog_manager.cc:1259] Loaded cluster ID: 16cd60c3d19446739734e5a74158ffd2
I20260923 19:35:13.013265 23424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac: loading cluster ID for follower catalog manager: success
I20260923 19:35:13.016059 23424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac: acquiring CA information for follower catalog manager: success
I20260923 19:35:13.017875 23424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3659d385e6524141b147735b66db15ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 19:35:13.135398 23445 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2980ee3690b44fb69ce5a0f1fe370487: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 19:35:13.371512 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:13.364501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.0.97.61:40553)
0923 19:35:13.365291 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:13.365299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:13.365323 (+ 24us) server_negotiation.cc:407] Connection header received
0923 19:35:13.365409 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:13.365414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:13.365610 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:13.365754 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:13.366304 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.367198 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.368765 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.369281 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.369432 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:13.370042 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:13.370780 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:13.370946 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:13.371050 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 19:35:13.371154 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":481,"threads_started":1}
I20260923 19:35:13.385813 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:13.378943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.0.97.60:36395)
0923 19:35:13.379644 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:13.379649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:13.379680 (+ 31us) server_negotiation.cc:407] Connection header received
0923 19:35:13.379860 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:13.379864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:13.380057 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:13.380259 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:13.380751 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.381621 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.383121 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.383574 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.383679 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:13.384310 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:13.385013 (+ 703us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:13.385141 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:13.385297 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 19:35:13.385385 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":386,"threads_started":1}
I20260923 19:35:13.400696 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:13.392507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.0.97.60:36395)
0923 19:35:13.392680 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:13.392685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:13.394857 (+ 2172us) server_negotiation.cc:407] Connection header received
0923 19:35:13.395114 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:13.395120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:13.395214 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:13.395302 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:13.395777 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.396551 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.398627 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.399251 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.399369 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:13.399820 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:13.399936 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:13.400015 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:13.400236 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 19:35:13.400354 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 19:35:13.402096 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:13.392464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.0.97.62:36317)
0923 19:35:13.393329 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:13.393335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:13.394032 (+ 697us) server_negotiation.cc:407] Connection header received
0923 19:35:13.394351 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:13.394357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:13.394617 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:13.395588 (+ 971us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:13.395778 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.397361 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.399173 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.399859 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.399999 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:13.400701 (+ 702us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:13.401139 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:13.401625 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:13.401678 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 19:35:13.401778 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":602,"threads_started":1}
I20260923 19:35:13.402573 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 19:35:13.392634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.0.97.61:40553)
0923 19:35:13.392851 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 19:35:13.392857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 19:35:13.395209 (+ 2352us) server_negotiation.cc:407] Connection header received
0923 19:35:13.395574 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 19:35:13.395582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 19:35:13.395700 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 19:35:13.395779 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 19:35:13.397253 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.398439 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.400508 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 19:35:13.401102 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 19:35:13.401191 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 19:35:13.401781 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 19:35:13.401905 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 19:35:13.402005 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 19:35:13.402183 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 19:35:13.402270 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 19:35:13.407333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 23163
I20260923 19:35:13.408653 23225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:35:13.522324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 23163
W20260923 19:35:13.543792 23329 connection.cc:535] server connection from 127.0.0.1:45278 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 19:35:13.544581 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 23237
I20260923 19:35:13.546000 23299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:35:13.665678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 23237
I20260923 19:35:13.683796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu pid 23311
I20260923 19:35:13.685128 23373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 19:35:13.805441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSGBcM/build/asan/bin/kudu with pid 23311
2026-09-23T19:35:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4900 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9621 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (369484 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST